Stock help page for DVID server currently running on e06u10.int.janelia.org

Welcome to DVID

This page provides an introduction to the currently running DVID server. Developers can visit the Github repo for more documentation and code. The DVID admin console may be available if you have downloaded the DVID console web client repo and included -webclient=/path/to/console when running the dvid serve command.

HTTP API and command line use

All API endpoints (except for memory profiling) follow this layout: 

/serverhost:someport/api/...
The online documentation doesn't show the server host prefixed to the "/api/..." URL, but it is required.

General commands

 GET  /api/help

	The current page that lists all general and type-specific commands/HTTP API.

 GET  /api/help/{typename}

	Returns help for the given datatype.

 GET  /api/load

	Returns a JSON of server load statistics.

 GET  /api/storage

 	Returns a JSON object for each backend store where the key is the backend store name.
	The store object has local instance ID keys with the following object value:

	{
		"Name": "grayscale",
		"DataType": "uint8blk",
		"DataUUID": ...,
		"RootUUID": ...,  // this is the UUID of this data's root
		"Bytes": ...
	}

 GET  /api/heartbeat[?u=]

	Preferred method to test whether server is alive.  If a username is provided, the
	time it takes to respond to the request (including transmission to remote client)
	is recorded.  A compilation of user latencies is available through the 
	/api/user-latencies endpoint.

GET /api/user-latencies[?u=,
			"LastLatency": ,
			"MinLatency": ,
			"MaxLatency": 
		},
		...
	]

GET  /api/server/info

	Returns JSON for server properties.

GET  /api/server/note 

	Returns any value of [server.note] from the configuration TOML.

GET  /api/server/config 

	Returns the raw content of the configuration TOML.

GET  /api/server/types

	Returns JSON with the datatypes of currently stored data instances.  Datatypes are represented
	by a name and the URL of the reference implementation.  To see all possible datatypes, i.e., the
	list of compiled datatypes, use the "compiled-types" endpoint.

GET  /api/server/compiled-types

 	Returns JSON of all possible datatypes for this server, i.e., the list of compiled datatypes.

GET  /api/server/groupcache

 	Returns JSON for groupcache statistics for this server.  See github.com/golang/groupcache package
	Stats and CacheStats for MainCache and HotCache.

GET /api/server/blobstore/{reference}
   
	GETs data with the given reference string from this server's blobstore. The blobstore is
	populated as part of mutation logging and is read-only.  The reference is a URL-friendly 
	content hash (FNV-128) of the blob data.

---- Server endpoints that require use of additional authorization by requester ----

POST  /api/server/settings

	Sets server parameters.  Expects JSON to be posted with optional keys denoting parameters:
	{
		"gc": 500,
		"throttle": 2
	}

	 
	Possible keys:
	gc        Garbage collection target percentage.  This is a low-level server tuning
				request that can affect overall request latency.
				See: https://golang.org/pkg/runtime/debug/#SetGCPercent

	throttle  Maximum number of CPU-intensive requests that can be executed under throttle mode.
				See imageblk and labelblk GET 3d voxels and POST voxels.
				Default = 1.


POST /api/server/reload-auth

	Reloads any authorization file as configured in the TOML file.

POST /api/server/reload-blocklist

	Reloads any blocklist file as configured in the TOML file.


-------------------------
Memory Profiler endpoints
-------------------------

 GET  /profiler/start

 	Starts a memory profiler and redirects to a web page where you can see real-time
	memory usage graphs.

 GET  /profiler/stop

 	Stops the memory profiler.

 GET  /profiler/info.html

 	Shows real-time memory usage graphs.

 GET  /profiler/info

 	Returns JSON of memory usage data.


-------------------------
Repo-Level REST endpoints
-------------------------

 POST /api/repos

	Creates a new repository.  Expects configuration data in JSON as the body of the POST.
	Configuration is a JSON object with optional "alias", "description", "root" (the desired
	UUID of the root), and "passcode" properties. Returns the root UUID of the newly created 
	repo in JSON object: {"root": uuid}

 GET  /api/repos/info

	Returns JSON for the repositories under management by this server.
	Note that any versioned properties for image-based data instances (e.g., extents) 
	will be drawn from the leaf of the master branch.

 HEAD /api/repo/{uuid}

	Returns 200 if a repo with given UUID is available.

 GET  /api/repo/{uuid}/info

	Returns JSON for just the repository with given root UUID.  The UUID string can be
	shortened as long as it is uniquely identifiable across the managed repositories.
	Note that any versioned properties for image-based data instances (e.g., extents) 
	will be drawn from the leaf of the master branch.

 POST /api/repo/{uuid}/info

	Allows changing of some repository properties by POSTing of a JSON similar to what
	you'd use in posting a new repo.  The "alias" and "description" properties can be
	optionally modified using this endpoint by POST of a JSON like:
	{
		"alias": "myrepo",
		"description": "This is the best repository in the universe"
	}
	Leaving out a property will keep it unchanged.

 POST /api/repo/{uuid}/instance

	Creates a new instance of the given data type.  Expects configuration data in JSON
	as the body of the POST.  Configuration data is a JSON object with each property
	corresponding to a configuration keyword for the particular data type.  

	JSON name/value pairs:

	REQUIRED "typename"     Type name of the new instance,
	REQUIRED "dataname"     Name of the new instance
	OPTIONAL "versioned"    If "false" or "0", the data is unversioned and acts as if
							all UUIDs within a repo become the root repo UUID.  (True by default.)
	OPTIONAL "Compression"  Specify the compression format to use when serializing values to disk.
							(Applies to most instance types, but not all.)
							Choices are: “none”, “snappy”, “lz4”, “gzip”, “jpeg”.
							Where applicable, the compression level can be appended, e.g. "jpeg:80".
	OPTIONAL "Tags"         Can send list of tags as a series of equal statements separated by
							commas, e.g., "type=meshes,stuff=something-something".  This will
							create a tag "type" set to "meshes" and a tag "stuff" set to 
							"something-something".  Note that the formatting prevents spaces
							and commas from being part of a tag.

	A JSON message will be sent to any associated Kafka system with the following format:
	{ 
		"Action": "newinstance",
		"UUID": ,
		"Typename": ,
		"Dataname": 
	}
					  	  

  GET /api/repo/{uuid}/log
 POST /api/repo/{uuid}/log

	GETs or POSTs log data to the repo with given UUID.  The get or post body should 
	be JSON of the following format: 

	{ "log": [ "provenance data...", "provenance data...", ...] }

	The log is a list of strings that will be appended to the repo's log.  They should be
	descriptions for the entire repo and not just one node.  For particular versions, use
	node-level logging (below).

  GET /api/repo/{uuid}/branch-versions/{branch name}

	Returns a JSON list of version UUIDs for the given branch name, starting with the
	current leaf and working back to the root.  Use "master" for the default branch.

	If a repository predates the introduction of branch names and has multiple paths
	for the given branch name, the endpoint will return an Bad Request Error (400).

 POST /api/repo/{uuid}/merge

	Creates a conflict-free merge of a set of committed parent UUIDs into a child.  Note
	the merge will not necessarily create an error immediately, but later GETs that
	detect conflicts will produce an error at that time.  These can be resolved by
	doing a POST on the "resolve" endpoint below.

	The post body should be JSON of the following format: 

	{ 
		"mergeType": "conflict-free",
		"parents": [ "parent-uuid1", "parent-uuid2", ... ],
		"note": "this is a description of what I did on this commit"
	}

	The elements of the JSON object are:

		mergeType:  must be "conflict-free".  We will introduce other merge options in future.
		parents:    a list of the parent UUIDs to be merged. 
		note:       any note that should be set for the child version.

	A JSON response will be sent with the following format:

	{ "child": "3f01a8856" }

	The response includes the UUID of the new merged, child node.

 POST /api/repo/{uuid}/resolve

	Forces a merge of a set of committed parent UUIDs into a child by specifying a
	UUID order that establishes priorities in case of conflicts (see "parents" description
	below.  

	Unlike the very fast but lazily-enforced 'merge' endpoint, this request spawns an 
	asynchronous routine that checks all data for the given data instances (see "data" in
	JSON post), creates versions to delete conflicts, and then performs the conflict-free 
	merge to a final child.  

	The post body should be JSON of the following format: 

	{ 
		"data": [ "instance-name-1", "instance-name2", ... ],
		"parents": [ "parent-uuid1", "parent-uuid2", ... ],
		"note": "this is a description of what I did on this commit"
	}

	The elements of the JSON object are:

		data:       A list of the data instance names to be scanned for possible conflicts.
		parents:    A list of the parent UUIDs to be merged in order of priority.  If 
					 there is a conflict between the second and third UUID, the conflicting
					 data in the third UUID will be deleted in favor of the second UUID.
		note:       Any note that should be set for the child version.

	A JSON response will be sent with the following format:

	{ "child": "3f01a8856" }

	The response includes the UUID of the new merged, child node.


-------------------------
Node-Level REST endpoints
-------------------------

Note: UUIDs referenced below are strings that may either be a unique prefix of a
hexadecimal UUID string (e.g., 3FA22) or a branch leaf specification that adds
a colon (":") followed by the case-dependent branch name.  In the case of a
branch leaf specification, the unique UUID prefix just identifies the repo of
the branch, and the UUID referenced is really the leaf of the branch name.
For example, if we have a DAG with root A -> B -> C where C is the current
HEAD or leaf of the "master" (default) branch, then asking for "B:master" is
the same as asking for "C".  If we add another version so A -> B -> C -> D, then
references to "B:master" now return the data from "D".


  GET /api/node/{uuid}/note
 POST /api/node/{uuid}/note

	GETs or POSTs note data to the node (version) with given UUID.  The get or post body should 
	be JSON of the following format: 

	{ "note": "this is a description of what I did on this version" }

	The log is a list of strings that will be appended to the node's log.  They should be
	data usable by clients to reconstruct the types of operation done to that version
	of data.

	A JSON message will be sent to any associated Kafka system with the following format:
	{ 
		"Action": "nodenote",
		"UUID": ,
		"Note": 
	}


  GET /api/node/{uuid}/log
 POST /api/node/{uuid}/log

	GETs or POSTs log data to the node (version) with given UUID.  The get or post body should 
	be JSON of the following format: 

	{ "log": [ "provenance data...", "provenance data...", ...] }

	The log is a list of strings that will be appended to the node's log.  They should be
	data usable by clients to reconstruct the types of operation done to that version
	of data.

	A JSON message will be sent to any associated Kafka system with the following format:
	{ 
		"Action": "nodelog",
		"UUID": ,
		"Log": [, , ...]
	}


 GET /api/node/{uuid}/status

	Returns the commit or lock status of the node with given UUID in JSON format:

	{ "Locked": true }

 POST /api/node/{uuid}/commit

	Commits (locks) the node/version with given UUID.  This is required before a version can 
	be branched or pushed to a remote server.  The post body should be JSON of the 
	following format: 

	{ 
		"note": "this is a description of what I did on this commit",
		"log": [ "provenance data...", "provenance data...", ...] 
	}

	The note is a human-readable commit message.  The log is a slice of strings that may
	be computer-readable.

	If successful, a valid JSON response will be sent with the following format:

	{ "committed": "3f01a8856" }

	A JSON message will be sent to any associated Kafka system with the following format:
	{ 
		"Action": "commit",
		"UUID": 
		"Note": ,
		"Log": [, , ...]
	}


 POST /api/node/{uuid}/branch

	Creates a new branch child node (version) of the node with given UUID.
	The branch name must be unique across the DAG.

	The post body should be in JSON format, where "note" and "uuid" are optional:

	{
		"branch": "unique name of new branch",
		"note": "this is what we'll be doing on this version",
		"uuid": 
	}

	A JSON response will be sent with the following format:

	{ "child": "3f01a8856" }

	The response includes the UUID of the new child node.

	A JSON message will be sent to any associated Kafka system with the following format:
	{ 
		"Action": "branch",
		"Parent": 
		"Child": ,
		"Branch": ,
		"Note": 
	}

	
 POST /api/node/{uuid}/newversion

	Creates a new child node (version) of the node with given
	UUID if no open node exists.

	An optional post body should be in JSON format:

	{ 
		"note": "this is what we'll be doing on this version",
		"uuid": 
	}

	A JSON response will be sent with the following format:

	{ "child": "3f01a8856" }

	The response includes the UUID of the new child node.

	A JSON message will be sent to any associated Kafka system with the following format:
	{ 
		"Action": "newinstance",
		"Parent": 
		"Child": ,
		"Note": 
	}


 POST /api/node/{uuid}/tag

	Tags a version with a unique string across the DAG (including UUIDs).

	The post body should be in JSON format, where the tag name is specified
	in the "tag" field and a "note" is optional:

	{
		"note": "this is what this tag means",
		"tag": "my-great-tag"
	}

	A JSON message will be sent to any associated Kafka system with the following format:
	{ 
		"Action": "tag",
		"Parent": 
		"Child": ,
		"Branch": "tag-",
		"Note": "Tag of version  with "
	}

 GET /api/node/{uuid}/{data name}/blobstore/{reference}
 POST /api/node/{uuid}/{data name}/blobstore

	GETs or POSTs data into a blob store associated with the given data instance.
	The reference is a URL-friendly content hash (FNV-128) of the blob data.
	The POST will store data and return the following JSON:

	{ "reference": "" }

	Note that POST /blobstore will not be logged in any associated kafka system.

Data type commands

This server has compiled in the following data types, each of which have a HTTP API. Click on the links below to explore each data type's command line and HTTP API.

Background batch processes like generation of tiles, sparse volumes, and various indices, will be paused if a single server receives more than a few data type API r requests over a 5 minute moving window. You can mark your API request as non-interactive (i.e., you don't mind if it's delayed) by appending a query string interactive=false.

Licensing

DVID is released under the Janelia Farm license, a 3-clause BSD license.