Transactions and Concurrency in Postgresql

Yesterday, I was in a event, talking about Transactions, ACID, Isolation phaenomena, etc., under Postgresql.

In Postgresql or any other DBMS, Transactions might be running in sequence (serializable) or concurrently, depending on the Isolation level set (they are implemented by DBMS vendors in their own way).


The exagerated simple database table presented above, its two records of bank account holders and their balances, will be used for this article.

The focus of the discussion is about concurrent Transactions with READ COMMITED Isolation Level (default in Postgresql), but also demonstrating two transactions running, one SERIALIZABLE, and the other READ COMMITED.



READ COMMITED vs READ COMMITED


Two transactions were started. Observe the CLOCK icons on PgAdmin, indicating that a Transaction is in progress (on each Query Tool window)...


On the left: the balance of Steve Jobs was already updated to 1100

On the right: Steve Jobs still have 1000 of balance. Why?

Postgresql default Isolation level is READ COMMITED, which does not allow DIRTY READ phaenomena (data which is not commited yet by one Transaction, is visible to other Transactions).



Adding More Money


On the left: the Transaction didn't finished, yet.

On the right: more money to Steve Jobs account, but the query doesn't finish...

The database table accounts is locked by the Transaction on the left. Only when this Transaction finishes (COMMIT), is when the Transaction on the right will add 400 bucks more on Steve Jobs acccount.


On the left: Transaction has finished and 100 bucks were deposited to Steve Jobs account

On the right: still in progress, but since the Transaction on the left finished (table released from lock), then the Transaction on the right sees a balance of 1100, adding 400 to it.

But why data commited from one Transaction is visible to another Transaction that is in progress?

On Isolation level of READ COMMITED, a Transaction can read data that was COMMITED. That being said, one phaenomena that might happen is what is called NON-REPEATABLE READ, where changes commited by one Transaction in a column, are VISIBLE to Transactions with Isolation Level of READ COMMITED.

If these Transactions read the data again, they might get a different value (NON-REPEATABLE), which might lead to some data inconsistencies during READ and WRITE operations of concurrent Transactions, of course, without affecting the ACID property of "Consistency" (transactions must respect database constraints - e.g. balance >= 0).


At the end, Transaction on the right has finished, adding 400 bucks more on Steve Jobs account, leaving the account with a balance of 1500 on the left, where no Transaction is in progress.



SERIALIZABLE vs READ COMMITED


Just as before, two Transactions in progress, being the first with Isolation Level SERIALIZABLE.


Records are still the same since the previous example.


Just as the specification, the PHANTOM READ phaenomena doesn't happen with a Transaction running with Isolation Level SERIAZABLE...


...even when the READ COMMITED Transaction finishes.

Doesn't matter: SERIAZABLE Transaction will see what's happening on it's snapshot of accounts table, and nothing else that is happening in other Transactions.



Some Reflections

Weaker isolation levels promote more performance (less overhead) at the cost of temporary data inconsistency.

It really depends on the Business Rules that govern the design of a Backend service, to determine which Isolation Level best suits. Maybe a PHANTOM READ or a NONREPETABLE READ might not cause any trouble in terms of UX for an application that is performing operations on the Stock Market (prices fluctuating from seconds to seconds), but maybe for an E-commerce solution, having a price changed all of sudden for a product that is current under a sell/buy transaction with an User, is not the best scenario: would be bad to start the acquisition of a product that costs U$ 100.00, and in the middle of the transaction (before paying), the price increases to U$ 125.00.

If you have any questions, I'd be glad to address them.

Your Repository of Images at Docker Hub

I just want to have a Docker Image of Postgresql on Docker Hub, with preloaded data saved, so I can pull it and create a container whenever I want, even sharing the image with other people for testing general stuff (just as a code repo on Github). I also want to update the preloaded data with data that I'm currently working on, for personal projects (never including sensitive data).

Hope these instructions might help you too, Developer or SysAdmin.



Requirements

Of course, having a Docker Hub account, with Docker engine installed and Docker client authenticated, are necessary in order to work with Docker. But besides that, without having a image repo created on Docker Hub, there's no place to upload locally built images.

This is the repo which I created:





Postgresql-based Container

First, I need to create a container based on Postgres official image repo (version 12).

I used to be more detail-oriented on creating containers (docker pull, docker create and docker start), but for the general operations, docker run is way more practical:

$ docker run -p 5432:5432 -e POSTGRES_PASSWORD=postgres -d postgres:12
Unable to find image 'postgres:12' locally
12: Pulling from library/postgres
a803e7c4b030: Pull complete 
89aefdc7e2ee: Pull complete 
9ed34906f4b2: Pull complete 
731d5308799a: Pull complete 
91955551cb08: Pull complete 
f3e0af904c74: Pull complete 
906febd4182a: Pull complete 
e7f594a72a36: Pull complete 
0857ffe75e3a: Pull complete 
29faa49de6a6: Pull complete 
de18a1e43334: Pull complete 
caec1f1d9715: Pull complete 
c75a666c6c4d: Pull complete 
Digest: sha256:81ab067d5573cdd38f6dac3de637aa1d6e923c3504bd93a79f1fb397675342f4
Status: Downloaded newer image for postgres:12
22f9858b88b5c8fca20bf7263aca18a4335836b6519bdebfa50d99823979113d




Creating Some Data

By the way, I'm taking a course on advanced DBMS topics, and for the tests that I need right now, this is enough:

$ psql -U postgres -h 127.0.0.1
Password for user postgres: 
psql (12.16 (Ubuntu 12.16-0ubuntu0.20.04.1))
Type "help" for help.

postgres=# CREATE TABLE accounts (id SERIAL PRIMARY KEY UNIQUE NOT NULL, holder VARCHAR(64) NOT NULL, balance INTEGER NOT NULL);
CREATE TABLE
postgres=# INSERT INTO accounts (holder, balance) VALUES ('Dennis Ritchie', 1000), ('Steve Jobs', 1000);
INSERT 0 2
postgres=# SELECT * FROM accounts;
 id |     holder     | balance 
----+----------------+---------
  1 | Dennis Ritchie |    1000
  2 | Steve Jobs     |    1000
(2 rows)

postgres=# exit




Preloaded Image with Dump

The custom Postgresql image that I'm going to create, will require this dump, in order to preload the containers (creating table + insert records):

$ pg_dump -U postgres -h 127.0.0.1 > postgres_db_dump.sql




Building Custom Image

I used postgres:12 image before, in order to build the container where the dump was generated. Since the image is locally cached, it will not require further download, while building the custom image.

Besides storing the dump in /docker-entrypoint-initdb.d/ directory (Postgres will execute the .sql files which are inside of it), the environment variable POSTGRES_PASSWORD will also be set during building level, in order to not require password of postgres user, while building containers based on this custom Postgres image:

$ cat << EOF > Dockerfile
> FROM postgres:12
> ENV POSTGRES_PASSWORD=postgres
> COPY postgres_db_dump.sql /docker-entrypoint-initdb.d/
> EOF

Building an image based on Dockerfile, requires the definition of a tag (version) of the image.

I'm gonna use the version of Postgresql engine (12), since later on, I might want to have another custom Postgresql, but for the newest version, say 16:

$ docker image build . -t ivanlmj/postgres:12
[+] Building 0.0s (7/7) FINISHED                                          
 => [internal] load build definition from Dockerfile                      0.0s
 => => transferring dockerfile: 140B                                      0.0s
 => [internal] load .dockerignore                                         0.0s
 => => transferring context: 2B                                           0.0s
 => [internal] load metadata for docker.io/library/postgres:12            0.0s
 => [internal] load build context                                         0.0s
 => => transferring context: 42B                                          0.0s
 => [1/2] FROM docker.io/library/postgres:12                              0.0s
 => CACHED [2/2] COPY postgres_db_dump.sql /docker-entrypoint-initdb.d/   0.0s
 => exporting to image                                                    0.0s
 => => exporting layers                                                   0.0s
 => => writing image sha256:a6e4657bdcaadf7536bead9eb....                 0.0s
 => => naming to docker.io/ivanlmj/postgres:12                            0.0s




Pushing to Repo

$ docker push ivanlmj/postgres:12
The push refers to repository [docker.io/ivanlmj/postgres]
3ab18e48bca6: Layer already exists 
f807adebe19a: Layer already exists 
19b47a8895e5: Layer already exists 
8b806c56785b: Layer already exists 
cf3b6ab8b6e9: Layer already exists 
c3d26035203f: Layer already exists 
9bb81d691b8a: Layer already exists 
572272d8013b: Layer already exists 
78f5cd522124: Layer already exists 
4e9de6962b22: Layer already exists 
6722c8542886: Layer already exists 
20fde8a4b2e7: Layer already exists 
8fb3f4d77297: Layer already exists 
d310e774110a: Layer already exists 
12: digest: sha256:fa9cf0626b33882c2d71b4e8fffec87691b...  size: 3247




Container Based on Custom Image (testing)

Since the image was built with the environment variable with the password of postgres user, there's no need to set it when creating a container:

$ docker run -p 5432:5432 -d ivanlmj/postgres:12
b8de9890aed0600ba53cfbe5d3b4efdada6460b669b19373406cf36a379d621f

$ docker ps
CONTAINER ID   IMAGE                 COMMAND                  CREATED         STATUS        PORTS                                       NAMES
b8de9890aed0   ivanlmj/postgres:12   "docker-entrypoint.s…"   2 seconds ago   Up 1 second   0.0.0.0:5432->5432/tcp, :::5432->5432/tcp   admiring_mclean

Connecting via host client to published port 5432, works just fine, and the data instrcuted by the dump, is present on the container database:

$ psql -U postgres -h 127.0.0.1
Password for user postgres: 
psql (12.16 (Ubuntu 12.16-0ubuntu0.20.04.1))
Type "help" for help.

postgres=# SELECT * FROM accounts;
 id |     holder     | balance 
----+----------------+---------
  1 | Dennis Ritchie |    1000
  2 | Steve Jobs     |    1000
(2 rows)

Every time I see an individual supporting A.I. services, I just chuckle

The airplane was supposed to transport people.
Radioactivity was discovered for supporting health treatments.
And yet, we found "another utility" for both of them.

Lockheed Martin F-35 Joint Strike Fighter, on a simulation of nuclear bomb launch


A.I. is way more powerful than airplanes and radioactivity combined, in a world that is digitally unstoppable. In a world where there's no legal framework that could at least, define sanctions, prohibitions and punishment for the "bad" use of A.I., it's really bad, to use and support these technologies.

A.I. is now capable of drafting lawsuits, pass exams, and write entire web sites: is our society ready to deal with the economic, political and educational impact that this could cause? Are we ready for such a catastrophe?

As "The Godfather of A.I." who resigned today from Google said:

“The idea that this stuff could actually get smarter than people — a few people believed that. But most people thought it was way off. And I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that.” - Geoffrey Hinton

Steve Wozniak has also a word about the topic:
:


A.I. can automate machine gun turrets, increase brute force attacks exponentially, severely deepfake individuals just as it can diagnose a cancer on a MRI. You might see a dilema here, but with the current legal frameworks that we have, but truth be told: first, this is just going to be improved in order to fill the pocket of somebody else, and after that, it will bring bennefits to humankind.

The answer is pretty simple to me: without legal frameworks, the humanity is not ready for A.I.

"We want to help one another. Human beings are like that."

On the last 24hs, I helped different people, in a variety of ways.

A gentleman, not so savvy on social media, computers, etc., starting his business, asked me to help him to create a Facebook page, in order to promote a business that will help not just him, but also his family (including grandchildren).

My nephew, is quite confident of his talent and believe that without talent, you can't make things happen. He mentioned that he diminishes the efforts of his classmates that are training for soccer championships: I warned him that we must never, never, say that someone has no talent, that someone can't do something due to incapacity or anything related. He reflected upon my speech and understood that even Goku (yes, from Dragon Ball anime), as a powerful, strong and gifted warrior, had challenges, and by training and dedication, he got even better, more capable, more strong.

My mother, on her 60's, living emotional challenges on this life stage, missing my presence and the presence of my brothers, and it's my turn, to provide "a therapy session", working together, so she can ease her mind, understanding that we all love her, even though we are busy people.

A great friend of mine, suffered the impact of the current economic scenario that we are all living, being fired from a company where he devoted gigantic efforts. I believe that we both understood that this is just a phase, and that he's a great professional, who got impacted by this unstable economic scenario, where big techs are firing people, but making billionaire acquisitions of A.I. companies, at the same time (funny, isn't it?). He will be working for another great company, soon enough. Meanwhile, he will take his time to explore ideas and knowledge which are quite difficult to explore while working from Monday to Friday.

It came to my knowledge, the work of The Humane Society of the United States, rescuing dogs from farm meats in South Korea. It wouldn't hurt me to donate U$ 10 dollars. I'm no hero for that, for there are other organizations, working for humanitarian and animal causes, but it made me feel good about helping at least one of the causes that I would like to support.

In this world, with so many challenges and obstacles, the least that we can do, is to give a hand to those who need our abilities, capacity, intelligence, patience, and even resources.

As my final words, here I quote a line from the movie "The Great Dictator" (1940), where Charlie Chaplin says:

"We want to help one another. Human beings are like that. We want to live by each other's happiness, not by each other's misery."

7 years of StackOverflow

Seems like yesterday...

I was once banned from asking questions, for 1 year. At the beginning, I asked a bunch of stupid questions like "How to create an array of strings in Java?" or "What is the on this array construction?": Senior Moderators couldn't cope with it.

Can't fully blame them (although they were mean): the questions were about basic and specifiic programming language concepts and not real software problems.


At that time, I was an underdog, rejected on many job interviews (most of them without any feedback, as usual), plus these Moderators, banning my participation on asking questions. Living abroad. I felt bad.

I studied as much as I could, in order to become better, asking and writing clear problem statements (after the ban was lifted), providing the best answers I could.

Being Moderator and Contributor of StackOverflow, even with rude and mean people out there, was tough, but it really helped me to build my career. Writing better documentation, interacting with other contributors, without being mean, reasonating about issues, solutions, putting clear statements for a problem or answer.

All can I say is "thank you", even to the bad people in the SO community: thankfully, there are more good than bad people.

Mastodon