iamcosban

Getting Started with Persistence ORM for Go

Persistence is a useful ORM and query builder/executor package for the Go programming language. It aims to remain dialect agnostic, easy to use, and to not hide functionality from the user. Like many of my other software projects, it was built due to the frustration I had with other similar libraries available at the time. Absolutely no claims could be made that persistence is the best solution for this problem set at the moment, but it is definitely good enough tool. All of my projects which interact with sql databases use it though one could bet that this is probably just me eating my own dog food.

Installation

Just install it with go get for most dialects 1

go get -u gitlab.com/cosban/persistence

Selecting a Database

Given that persistence tries to be as dialect agnostic as possible, it makes no effort to choose a default dialect for you to use. You will need to import one that matches the database you plan on connecting to. Luckily, drivers for both postgres and sqlite3 come included.

// postgres
import _ "gitlab.com/cosban/persistence/postgres"

// sqlite3
// import _ "gitlab.com/cosban/persistence/sqlite3"

// some other driver
// import _ "your.git/repo"

Only one dialect may be chosen at a time because there has not yet been a use-case for requiring multiple dialects at the same time.

Connecting

Connecting to the database is as straight-forward as importing the dialect, creating a connection string for that dialect, and actually connecting. Connection strings need to be formatted in a way specified by the underlying database.

package main

import (
    "gitlab.com/cosban/persistence"
    _ "gitlab.com/cosban/persistence/sqlite3" // for an in memory database
)

func main() {
    c, err := persistence.Open("sqlite3", "file::memory:?mode=memory")
    // or "postgres" for the postgres driver
    if err != nil {
        panic("unable to connect to the database")
    }
    defer c.Close()
}

Models

Data which should be mapped to entries within a database are modeled as structs or pointers to structs. Only fields which are exported will be used as columns within the database

Here is an example of a simple model:

type Book struct {
    ID        uint64     `persistence:"primary,generated"` // a primary key that auto increments
    Title     string     `persistence:"unique,not_null"`   // require uniqueness and non-nullness
    Published *time.Time `persistence:"generated"`         // automatically insert now() as the date
    Author    string
    Ignored   int        `persistence:"-"`                 // ignore this field
}

Model Tags

In order to make well-formed tables, it may be required that constraints, default values, and other attributes are put into the database schema. Persistence does this through struct optional tags.

The following are supported for use:

Tag Details
name Used to override the default column name
not_null Used to specify that a column must have a NOT NULL constraint
unique Used to specify that a column must have a UNIQUE constraint
primary Used to specify that one or more columns is part of a PRIMARY KEY
primary_key Used to specify that one or more columns is part of a PRIMARY KEY
combined Used to specify that one or more columns is part of a composite key (unique)
combined_key Used to specify that one or more columns is part of a composite key (unique)
default Used to specify the DEFAULT value of a column
references Used to specify that this column is a FOREIGN KEY on table
foreign Used to specify that this column is a FOREIGN KEY on table
foreign_key Used to specify that this column is a FOREIGN KEY on table
type Used to override the default type which the driver would otherwise choose
join Used to specify that a field is to be populated by a JOIN and is not modeled in this table
many_to_many Used to specify that a field is to be populated by a JOIN using a join table which allows for many-to-many relationships
generated Used to specify that a fiels should be automatically generated by the database. Each dialect driver determines how generated fields should be handled

Basic CRUD Usage with the query builder

package main

import (
    "time"

    "gitlab.com/cosban/persistence"
    _ "gitlab.com/cosban/persistence/sqlite3" // for an in memory database
)

type Book struct {
    ID        uint32    `persistence:"primary,generated"`
    Title     string    `persistence:"not_null"`
    Author    string    `persistence:"not_null"`
    Published time.Time `persistence:"generated"`
}

func main() {
    c, err := persistence.Open("sqlite3", "file::memory:?mode=memory")
    if err != nil {
        panic("unable to connect to the database")
    }
    defer c.Close()
    
    // create or update a table
    err = c.BuildStatement().CreateOrUpdateTable(Book{})
    if err != nil {
        panic("unable to create or update table")
    }

    // create/insert a new entry
    err = c.BuildStatement().Insert(Book{Title: "War and Peace", Author: "Leo Tolstoy"})
    if err != nil {
        panic("unable to insert book")
    }

    // read single item with id of 1
    var book Book
    err = c.BuildStatement().From(Book{}).Where("id = ?", 1).Query(&book)
    if err != nil {
        panic("unable to read book")
    }

    // update the book which was read
    book.title = "Война и миръ"
    err = c.BuildStatement().Update(book)
    if err != nil {
        panic("unable to update book")
    }

    // delete
    err = c.BuildStatement().Delete(book)
    if err != nil {
        panic("unable to delete book")
    }
}

Basic Raw Statement Execution

Given that many advanced queries are unable to be done with the query builder, Persistence allows the user to create and execute manually rolled prepared statements with the same connection used by the query builder.


package main

import (
    "time"

    "gitlab.com/cosban/persistence"
    _ "gitlab.com/cosban/persistence/sqlite3" // for an in memory database
)

type Book struct {
    ID        uint32    `persistence:"primary,generated"`
    Title     string    `persistence:"not_null"`
    Author    string    `persistence:"not_null"`
    Published time.Time `persistence:"generated"`
}

func main() {
    c, err := persistence.Open("sqlite3", "file::memory:?mode=memory")
    if err != nil {
        panic("unable to connect to the database")
    }
    defer c.Close()


  1. If your app relies on a sqlite3 database, you will need to ensure cgo is set up correctly before installing.

Reporting Accurate Unit and End-to-End Test Coverage in Golang

I write a lot of code in go. It has a nice testing suite, but one thing that has frustrated me is that I had previously been unable to generate an accurate coverage report for my unit and end to end tests. Sleuthing through other users’ solutions had been unsuccessful my use case, so I will document my findings here. The specific problem is that I want to include my package isolated end-to-end test coverage in with my unit test coverage report. After working and refining the process for which tests are run and coverage reports are produced, I have finally discovered a solution which works. My go query builder/ORM tool, persistence, will be the code-base I use to in this example and the end result will generate a badge like this: Coverage Report

prerequisites

Ensure the following are installed and in your path

  • go
  • make
  • tail

Golang has its own build/test commands which work great. With the amount of work I have in gitlab’s CI, it’s easiest to just specify that a make command should be run and have that command do all of the work.

code

The first step is to actually create an end to end test. This example does so in an e2e package which imports the root package. It is a separate package from the root package so that the project may be compiled without including any end to end tests which may make use of external packages, or packages that are not necessary for the main project to function.

package e2e

import (
        "gitlab.com/cosban/persistence"
        "testing"
)

func TestOneToManyJoin(t *testing.T) {
	if testing.Short() { // skip if -short flag is used
		t.Skip("skipping e2e test")
	}
        // actual testing occurs below
        // ...
}

Next, create a makefile. The following is a stripped down version with only the properties necessary for this example, the full makefile is available in the repository.

#makefile
PROJECT := persistence
PKG := "gitlab.com/cosban/$(PROJECT)"

.PHONY: test clean

test: clean
	@GO111MODULE=on go test ./... -short -coverprofile coverage.log
	@GO111MODULE=on go test $(PKG)/e2e -coverpkg $(PKG) -coverprofile coverage.e2e.log
	@tail -n +2 coverage.e2e.log >> coverage.log
	@go tool cover -func=coverage.log

clean:
	@rm -f coverage*.log

explanation

  1. the test rule requires that the clean rule is run first
  2. end to end tests have been skipped in the first pass because the tests include a check for the -short flag
  3. the end to end tests are then run and is told to generate a report for its coverage against the root package
  4. tail is used to concatenate all but the first line of the coverage.e2e.log file into the coverage.log file
  5. the coverage.log file is then analyzed by the go coverage tool and a report is printed out

further adjustments

If one wanted to expand the coverage beyond just one package for their end to end tests, they could do so by running the test against a separate coverpkg. As of speaking, the -coverpkg=all method does not work for this use-case because it causes all packages to be imported by the test. This is undesirable for persistence because only one dialect driver package should be imported at a time.

integration with gitlab

Within a project, navigate to settings > CI / CD > Test coverage parsing and then enter the pattern which will be used to determine test coverage. I use total:\s+\(statements\)\s+\d+.\d+%.

Arguing in Bad Faith

The last thing I want to do is affix my public name to anything political. In saying this and noting to myself that this is only the second post I’ve publicly written, I have already realized that if I declare my belief that a particular person argues or is arguing in bad faith then other people are going to be inclined to believe that I lean one way or another on a particular issue. As I’ve already said in my first post, no one is looking anyway.

A definition the term

To argue in bad faith is to willfully engage in verbal conflict with a person who appears to have an opposing opinion in such a way that the merits of either viewpoint are disregarded.

A few additional points to think about:

  1. Arguing in bad faith can still occur during violent agreements
  2. Those using sign language can happily argue in bad faith if they so choose to

For those that already know about Logical Fallacies; to argue in bad faith is to knowingly apply a logical fallacy to your argument in order to win.

When does this happen?

I am more apt to notice this behavior when reading or listening to political discussions. I’m sure that you will find people practicing this at other times though.

ok but give me some examples

I’m getting there: but first, a promise. I will never use the terms “Let’s talk about” or “We need to talk about” unless I am directly mentioning how I will never use those terms. I find them to be unnecessarily snarky and belittling to the reader.

With that, I’ll get into specific examples.

“Mr. Cohen committed four- four distinct federal crimes over a period of several years. He was motivated to do so by personal greed and repeatedly- repeatedly used his power and influence for deceptive ends, but the democrats don’t care. They don’t care, they just want to use you Mr. Cohen. You’re their patsy today. They’ve got to find somebody somewhere to say something so they can try to remove the president from office.”

Representative James Jordan from Ohio’s 4th District 2019-02-27

In James Jordan’s opening remarks for the public Michael Cohen hearing he attempts to persuade the public in multiple ways. Firstly, he attacks Michael Cohen’s credibility. I would not immediately consider this to be an example of arguing in bad faith due to the fact that Mr. Cohen had previously been proven to have lied to congress under oath. For the sake of Michael Cohen, one would hope that the evidence which was brought to the hearing is definitively able to prove the truth in his testamony. If, after being presented with admissable evidence, Mr Jordan were to continue to attack Michael Cohen, this would be proof that he is arguing in bad faith for that particular instance.
Representative Jordan absolutely argues in bad faith when he begins to speak about the motives from the democrat representatives on the committee. I see the irony in calling someone out for arguing in bad faith by claiming that they have stated someone is arguing in bad faith. Unfortunately I’m not able to see into the mind of Jim Jordan, nor am I able to do so for the Democrat members of the House Judiciary Committee. It is likely that both sides of the aisle are committing this sin. I’m not a political expert and perhaps the democrats will do anything to remove the president from office. Until this has been proven, stating such as a fact is not a valid way to present an argument though.

“Mr. Sloan, why was the involved in the caging of children and financing the caging of children to begin with?”
“Since Wells Fargo financed the building of [The North Dakota Access] pipeline in an environmentally unstable way, why shouldn’t the bank be held responsible for financing the cleanup of the disasters from these projects”

Representative Alexandrea Ocasio-Cortez from New York’s 14th District 2019-03-12

In her attacks against the CEO of Wells Fargo, or rather, Wells Fargo itself, AOC appears to purposefully invoke false equivalence as well as the straw man. She is appealing more toward the concept that her argument is correct than toward the idea that she should need to prove it. Perhaps, instead, she could have framed the first statement in a way that asks why Wells Fargo allowed its investees to use their money to “put children in cages”. This way, she could have still enforced the point of which she believes it was bad that the bank was involved with an entity which treated children in this way without falsely stating that they were directly involved. It could have also led into an argument that if Wells Fargo did not know about these behaviors, then they should begin to provide such an oversite to the way that their investments are used to ensure their money is used in humane ways.
Likewise, by stating that Wells Fargo should be directly held responsible for financing the clean up of the Keystone Pipeline, she is neglecting the fact that such a project should be insured against such disasters. Wells Fargo should have also considered whether project was not insured when providing the money to TransCanada Corporation. If it was not, or if they did not have an actionable resolution for cleanup in the event of a spill, then Wells Fargo would need to be held accountable for bad lending practices. Perhaps this is what she was trying to get at in her argument. Like James Jordan did before her though, she used bad tacticts when relaying her message.

In conclusion to all of what I have said, I want to ensure recognition that working as a Representative or Senator is hard. There are many, many issues that one needs to be aware of in order to do the job and while doing so, it is vital that they are effective in convincing their fellow elected members, their consituents, and the remaining public that the implications of the solutions to the problems being debated are important to think about. I simply wish that the method in which this debating occured would lean harder on fact than emotional appeal.
Certainly just providing two examples is not enough proof to state that my observation is fact. I’m just too lazy to dig out other cases where this has happened. I’m sure that you, reader who does not exist, will be able to pick up on when this tactic is used when it is so obviously deployed in one of the future arguments you happen to partake in or observe.

on creating nebula

I have a pattern of hating any program that I could “easily” write myself. It ultimately means that I spend a lot of my time writing code for solutions that already exist in some form. In the past, this came to light when I created a chat plugin for the bukkit, which was, at the time, the de-facto choice for managing multiplayer minecraft servers. I called it suckchat and claimed that all of the other chat plugins sucked. Sure, mine did too, but I had created this one and was free to do whatever I wanted with it.

My frustration with other peoples’ tools has led me to my latest endeavor: a project which I have very descriptively named nebula. Boiled down, it’s probably a website management platform. The main goal of it is… nebulous. I now work on it as an experimental/hobby project. This post is the first time I have been able to use it as a way to show anything to the public which means I have finally shipped a minimum viable product. Getting here has been hard though. My hobby projects tend to become abandoned as I lose interest in them, their requirements tend to grow as I pull in dependencies, begin to hate them, and decide that I could write something better. I suspect that is not a unique problem to me.

the stack

This instance of the site is served behind an nginx reverse proxy. nebula itself acts as a hub for interaction between the plugins which are used. The plugins used to serve cosban.net are as follows:

  • magnetar: an application for serving dynamic and static web pages.
  • sol: a user registration/management plugin. All it does is handle the login and registration of users.
  • rover: used to send emails to users (especially for registration)
  • aurora: a user permission manager
  • stasis: the tool which allows me to write blog and wiki entries in markdown

All of these plugins depend on common interfaces which have been defined in my core library which should probably be broken up even more than it currently is. Maybe I’ll get to that.

Further behind the scenes is persistence which I have developed out of my frustration for other database interacting libraries. I use it as a query builder/ORM to interact with postgres.

Usually the software I write is available for use under the MIT license

future goals

  1. The entire project seems to not use a lot of resources. As of writing, it appears to be using 36KB in memory. It should probably remain lightweight in that way.
  2. It should be able to handle a high load. It’ll probably need to support distribution at some point in order to do this.
  3. persistence needs to become fully featured. It currently works for most of my use-cases, but not all of them.
  4. support for metrics gathering would also be nice as I currently have no way of auditing anything that happens.

Maybe now that this is in the public eye (to be fair, no one is looking anyway) I’ll be motivated to continue working on this. Even if a gradually more usable product is developed, over a long period of time, I will consider that a success.