Querying Neo4j Clusters

As with most databases, with Neo4j, if you want to query the database, it’s simple. You use a driver, create a connection, send a query, and get back some results. That’s all there is to it!

Behind the scenes though, if you’re working with a clustered database, there’s a whole lot more going on than that. To begin with, the database isn’t in a single place but is composed of multiple servers. In this article, we’ll explore how Neo4j Clusters work and how Neo4j drivers get your query executed.

Before describing the drivers, we need a brief overview of how Neo4j clusters work and what the cluster roles are. This, in turn, will help you understand what a driver is doing.

Neo4j Causal Clustering

A cluster is composed of three or more Neo4j instances that communicate with one another to provide fault-tolerance and high-availability using a consensus protocol (RAFT). In Neo4j clustering, each database has a perfect, complete copy of the entire database (the graph is not sharded). Each machine in the cluster has a “role.” It can either be the leader or a follower.

Cluster Roles

The leader is responsible for coordinating the cluster and accepting all writes. Followers help scale the read workload ability of the cluster and provide for high-availability of data. Should one of the cluster machines fail and you still have a majority, you can still process reads and writes. If your cluster loses the majority, it can only serve (stale) reads and has no leader anymore.

Optionally, you can have any number of caches in the form of read replicas. They are read-only copies of your database for scaling out read-query loads. They are not officially part of the cluster, but rather are “tag along” copies that get replicated transactions from the main cluster.

Topology Changes

In the lifecycle of a cluster, cluster roles are temporary, not things you configure. Suppose you have machines A, B, and C. If A fails, then the remaining nodes (B and C) will elect a new leader amongst themselves. When A restarts, later on, it will rejoin the cluster, but probably as a follower. So roles can change through the lifecycle of the cluster. There are various other reasons where everything is working fine, where the cluster might elect a new leader — and so by themselves, role changes are not a cause for concern.

Neo4j uses the RAFT consensus algorithm to coordinate the cluster. Quite a lot is published on that topic if you’d like to go deeper.

Neo4j Drivers

The Driver API consists of 4 key parts illustrated below. Whether talking to a Neo4j cluster or single instance, all of these concepts apply.

In the cluster world though, because we are talking to a group of machines, how the transactions get executed and where they go is the subject of how “Routing Drivers” work.

Routing Drivers

When you use one of the supported Neo4j drivers (Java, Javascript, Python, .Net, and Go), there is an option to use the bolt+routing protocol. You’ll know you’re using it because it’s in the URI of the connection string. For example, you can connect to bolt+routing://neo4j.myhost.com.

Tip:

For clusters, set up a single DNS record with multiple A entries, each pointing to the cluster members.

This way, all clients can connect to the same DNS address but may have different machines as points of entry depending on cluster state and what is up. You don’t have to do it this way; you can connect directly to any of the server member’s IP addresses, but this way is more flexible should query topology change.

If you’re connecting to any single host or address, the driver handles the routing smarts for you. The driver will first check if the database has routing capabilities, and if so will fetch that holds a full or partial snapshot of the read and write services available. After that moment, the initial host will not be used unless the driver loses contact with the cluster and has to re-initialize routing.

Routing Tables

The routing table holds a list of servers that provide ROUTE, READ, and WRITE capabilities. This routing information is considered volatile and is refreshed regularly.

Drivers refresh this regularly because the cluster topology could change according to runtime events (like a machine failing). Generally, the WRITE node is going to be the leader, and the READ nodes are going to be the followers in the cluster or read-replicas.

Important Tip

Read replicas do not participate in cluster topology management, and as such, they do not provide routing tables. Only core nodes provide routing tables. This may change in subsequent releases of Neo4j.

If you’d like to try this out for yourself, just execute this statement on any clustered Neo4j instance:

CALL dbms.cluster.routing.getRoutingTable({}) 
YIELD ttl, servers 
UNWIND servers as serverRETURN ttl, server.role, server.addresses;

and you can see your routing table:

This particular cypher query is only intended for internal use and may change in future versions of Neo4j, so don’t write your code to it; but it gives you the idea of what the driver is actually doing.

Advertised Addresses

Where did those addresses in the routing table example come from? These are set by the user in the neo4j.conf file, as dbms.connector.bolt.advertised_address. In this way, Neo4j knows what address to publish to the external world where it can be contacted (Configuration Reference).

Important Tip

The advertised address setting is crucial for external clients to know how to contact your cluster. Set it explicitly in neo4j.conf

Make sure it is an address that can be resolved by external clients (and not an internal private address, such as 192.168.0.5). A very common error with Neo4j connectivity is to fail to set this or set it to an internal private address and have external clients on the Internet fail to connect because they cannot figure out how to reach your database!

Connection Management

Once a routing table in place, the driver can manage a pool of connections to all of the different machines in the cluster. What the user sees is usually just the creation of sessions and running queries from those sessions. What’s actually inside of the driver looks more like a pool of connections to a set of machines (A, B, and so on). Sessions simply borrow and ride on top of connections as needed to execute queries.

These connections are handled separately so that if (for example) Server A goes away and all of those connections end up dying, you, at the session level, don’t necessarily need to know about that. A session can still borrow a different connection and your application can keep chugging. An ongoing statement execution would fail and automatically be retried by the driver.

Users execute queries by using sessions. Sessions are cheap objects to create (unlike connections, which are expensive to set up). Sessions also provide a logical construct for chaining work together in a way that is causally consistent, meaning that you can do a series of transactions where subsequent transactions are always reading the writes made by earlier transactions in the same session.

Query Routing

Now let’s say your driver sends a query to the database. I’ll use JavaScript as an example, but the same concepts apply in any language.

The first thing we do is pull a session from the driver. We then use that session to execute the query.

const session = driver.session();
session.readTransaction(tx => tx.run('MATCH (n) RETURN count(n) as value'))   
  .then(results => {       
    console.log(results.records[0].get('value'));   
  })   
  .finally(() => session.close());

How does the driver know where to send this query? In this code example, we have used an explicit transaction or transaction function, meaning that we told the driver we’re doing a readTransaction. We were given a tx Transaction object, and with that, we ran our query. Because it has the routing table and it knows you’re doing a read, it will probably end up sending this query to one of the nodes with the READ role. If we did a write transaction, it would be sent to any node in the routing table that has a role of WRITE (which is generally the cluster leader).

If the routing driver has more than one choice of where to send the query, it uses a “least connected” strategy, which helps avoid accidentally routing to a server that is presently too busy to answer in a timely way.

Auto-Commit Transactions

What if you don’t use explicit transactions? You can skip the transaction function and instead, you could write:

const session = driver.session();
session.run(
  tx => tx.run('CALL library.myProcedure()'))
   .then(results => {
       console.log(results.records[0].get('value'));
   })
   .finally(() => session.close());

In this case, the driver cannot tell if you are doing a read or a write and will send your transaction to the leader. This way, the transaction should succeed even if you haven’t told it whether it needs to do a read or a write.

Important Tip

Neo4j drivers do not parse your Cypher to determine whether you’re reading or writing. Always use explicit transactions to tell it whether it’s a read or a write.

This, in turn, helps it route the query effectively and get the best utilization out of your cluster. If you never used explicit transactions, you might be sending all of your query loads to the leader, beating it up while leaving the other machines in your cluster idle.

Conclusion

In a clustered deployment, Neo4j utilizes smart bolt+routing clients to dynamically discover and monitor cluster topology and route your queries to machines in the cluster using a “least connected” strategy. All of this is done transparently for you, so at the user level, all you’ll see is creating sessions, running queries, and consuming the results.

Getting maximum benefit out of this setup requires some understanding of how this operates, and the most important parts to keep in mind are the following:

• Proper configuration of your advertised address
• Use of explicit transactions in driver code
• Where possible, DNS setup to create a single DNS record that all clients can use to talk to any node in the cluster

For more about Neo4j routing drivers, consult the Neo4j Operations Manual.