The Software Scaffold

This article serves as a practical guide for developers to ensure their APIs meet industry standards and best practices. This article takes a checklist approach, simplifying the complexities of REST API design and guiding developers through crucial decisions, from choosing appropriate HTTP methods and status codes to securing endpoints and implementing effective error handling. By following this structured overview, developers can avoid common pitfalls and create services that are both scalable and easy to maintain.

Considering the YADVS application here are some concepts to be clarified before going further:

• Server – the machine (virtual or physical) that hosts the API and processes client requests. It runs the voting system software and listens for incoming requests.
• API – the voting system API defines how clients can interact with the server’s services, such as creating a poll, voting, or viewing results. It provides the interface for accessing these actions.
• Service – a logical sub-division of an API like the poll management service that handles creating and managing polls (or the voting service that handles vote casting and aggregation). Services run on servers and are accessed via API calls.
• Endpoint – a specific URL, such as https://api.example.com/polls, is an endpoint where clients can interact with the poll management service. This endpoint allows clients to create or manage polls.

Naming Resources

• Use plural nouns unless you have a really good reason to use singular. (widely adopted)
• Paths SHOULD be easy to read and MAY reflect the hierarchy of underlying models. Paths SHOULD NOT end with a trailing /. [IBM Cloud]
• Resource paths should be built like /collectionId/{resourceId}/subCollectionId/{subSesourceId} [Google Cloud]
• Various notations can be used for naming collections kebab, snake, camel.
  • Lowercase seems to be favorite.
  • Kebab notation is preferred for being easier to read.
  • Consistency is anyway more important than any of the individual choices.
• A REST resource URL could look like this:
  • https://server.name:443/service-bundle-name/api/v1/resource-collection-id:optional-custom-method/individual-resource-id

Conforming to HTTP semantics

Mapping HTTP Methods to CRUD Operations

Here is a compilation of the HTTP verbs attributes [MDN, IBM Cloud – Methods]:

Mapping standard methods:

Custom methods [Google Cloud] should use the following generic HTTP mapping (follow the link below to see some examples):

https://service.name/v1/some/resource/name:customVerb

Returning HTTP Status Codes

Some of the most frequently used HTTP status codes with the REST services are [IBM Cloud]:

• 200 OK
  • This code MUST be returned for successful requests not covered by a more specific 2xx code.
• 201 Created
  • This code MUST be returned when a new resource was successfully created in a synchronous manner.
  • When returning this code, the server MUST also return a Location header with the URI of the created resource.
• 202 Accepted
  • This code MUST be returned when the request will be processed asynchronously, and may be rejected when it’s actually processed.
  • A resource representing the status of the request SHOULD be included in the response, and the response Location header SHOULD include a URI where this same status may be retrieved so that the client may stay up-to-date.
  • Used in asynchronous operations [Microsoft Azure].
• 204 No Content
  • This code MUST be returned when the server successfully processed the request and is not returning any content.
  • This MAY be used for DELETE requests.
  • It MAY occasionally be used for PUT or POST requests where it is impractical to return a resource representation because of size or cost, or for GET or PUT requests where a resource exists but has an empty representation.
  • Used in case of empty sets in message bodies [Microsoft Azure]
• 206 Partial Content
  • This code indicates that the server is delivering only part of the resource.
  • Servers MUST use this when enabling the client to resume interrupted downloads, or to split a download into multiple simultaneous streams.
• 301 Moved Permanently
  • This code SHOULD be returned when a resource’s URI changes, in order to keep the previous URI from failing.
  • When this code is returned, the server MUST also include a Location header.
  • It indicates that the current request, and all future requests, should be directed to the URI provided in the Location header.
  • Paths SHOULD NOT end with a trailing /; if a client appends a trailing /, the server SHOULD respond with a 301 status code along with a Location header containing the correct URI.
• 304 Not Modified
  • This code MAY be returned to indicate that the previously-downloaded resource a client possesses is up-to-date, and there’s no need to resend.
  • This code MUST NOT be returned except in response to conditional requests.
  • Responses with a 304 status MUST NOT have a response body.
  • Return this in case of PUT or a PATCH and nothing is modified.
• 400 Bad Request
  • This code MUST be returned when a request cannot be processed due to client error (e.g., the request is malformed or too large) when a more specific code is not applicable.
• 401 Unauthorized
  • This code MUST be returned when a request needs to be authenticated to succeed or the credentials provided are invalid.
  • Responses with a 401 status code MUST also provide a valid WWW-Authenticate header.
• 403 Forbidden
  • This code SHOULD be returned when the client is not allowed to make the desired request.
  • This status SHOULD be returned if the client is properly authenticated but does not possess permission to perform the request, or if the client is not authenticated, but authentication would not affect the outcome.
• 404 Not Found
  • This code SHOULD be returned when the requested resource could not be found but may be available in the future.
• 405 Method Not Allowed
  • This code SHOULD be returned when the requested resource doesn’t support the request method.
  • When this code is returned the response MUST include the Allow header with the list of accepted request methods for the resource.
• 406 Not Acceptable
  • This code MUST be returned when the resource the client requested is not available in a format allowed by the Accept header the client supplied.
  • Used for example in media type API versioning approach.
• 409 Conflict
  • This code SHOULD be returned when the request cannot be processed because of a conflict between the request and the current client-controlled state in the system.
• 415 Unsupported Media Type
  • This code MUST be returned when the client sends a payload of a content type the server cannot accept.
• 500 Internal Server Error
  • This code MUST be returned when a fatal error caused by an unexpected condition occurs on the server and was not caused by the client.
• 503 Service Unavailable
  • This code SHOULD be returned when the server is temporarily unavailable because it is overloaded or down for maintenance.
  • The server MAY include a Retry-After header telling the client when it should try submitting the request again.

Versioning the API

Various methods can be used for API versioning [Microsoft Azure]:

• URI Path Versioning [IBM Cloud]:
  • Versions are specified as part of the URL path, e.g.:
    • https://api.example.com/v1/users .
  • Pros: Easy to implement and understand.
  • Cons: It can lead to breaking changes in the structure of the API and results in redundant endpoints.
• Query String Versioning:
  • The version is specified in the query string, e.g.:
    • https://api.example.com/users?version=1 .
  • Pros: Clean, does not change the URL structure, and can allow for more flexibility (e.g., switching versions for individual requests).
  • Cons: Some older web browsers and web proxies will not cache responses for requests that include a query string in the URI. This can degrade performance for web applications that use a web API and that run from within such a web browser.
• Header Versioning:
  • API versions are specified in HTTP headers, e.g.:
    • Custom-Header: api-version=1 .
  • Pros: Keeps the URL clean, aligns with content negotiation principles.
  • Cons: Harder to implement and can be less visible for debugging purposes compared to versioning in the URL.
• Media Type Versioning (Accept Header):
  • Versions are declared via the media type in the Accept header, e.g.:
    • Accept: application/vnd.example.v1+json .
  • Pros: Clear adherence to REST principles using content negotiation.
  • Cons: Can make it more complex for developers to test and requires more understanding of HTTP headers.

Filtering, Sorting, and Pagination

Filtering, Sorting, and Pagination are essential components of REST APIs, particularly when dealing with large datasets. By implementing these features, REST APIs become more efficient, user-friendly, and scalable:

• Filtering: Allows clients to specify criteria to narrow down the results returned by an API. This helps improve performance and makes it easier for clients to find the specific data they need. Filtering can be further refined into:
  • Selection – the process of extracting specific rows from a table based on certain conditions (WHERE clause).
  • Projection – the process of extracting specific columns from a table. It projects only the desired fields onto a new result set.
• Sorting: Enables clients to order the results based on specific fields. This can be helpful for presenting data in a meaningful way, such as sorting a list of products by price or a list of customers by last name.
• Pagination: Divides large datasets into smaller, more manageable pages. This prevents overwhelming clients with excessive data and improves performance.

It is important to have a consistent convention for the query string attributes. There are various sources to choose from:

• Goole references:
  • Filtering
  • Sorting Order
  • List Pagination
• IBM Cloud
  • Filtering
  • Sorting
  • Pagination
• Microsoft Azure
  • for a standardized general approach, it is worths considering OData.

A convenient standard could be:

• use actual column names as query parameter names for selection
  • name=John&surname=Doe – to select by name and surname
  • age[gt]=14 – to choose records with age > 14
  • description[contains]=green – to select all descriptions containing the word “green”
  • value[in]=alpha,beta,gamma – to select rows with values in {“alpha”, “beta”, “gamma”} set
  • etc.
• ~fields – for projection
  • ~fields=name,description,age – will only return the 3 fields specified
• ~sort – for sorting
  • ~sort=name,-age – to return the rows sorted ascendingly by name and also descendingly by age
• ~limit, ~offset, and/or ~page – for pagination
  • ~limit=20 – will indicate page size
  • ~offset=100 – will indicate the table index (0 based) of the first returned row
  • ~page=3 – will indicate the page identifying token – e.g. if ~limit=20 and ~page=3 then ~offset=60

Optimizing APIs with Common Design Patterns

A comprehensive list of common design patterns for REST APIs is compiled by [Google Cloud]. A selective enumeration: Long Running Operations (also [IBM Cloud]), List Sub-Collections, Get Unique Resource From Sub-Collection, ETags, or Bool vs. Enum vs. String.

Partial responses

For large binary resources it may be preferable to allow for sending partial responses [Microsoft Azure]. The communication between client and server is structured in the following way:

1. the client sends a HEAD request to the server:
   • HEAD https://server.com/yadvs/api/v1/polls/123/votes/chart HTTP/1.1
2. the server responds providing some resource metadata

   • HTTP/1.1 200 OK
     
     Accept-Ranges: bytes
     Content-Type: image/jpeg
     Content-Length: 10922

3. the client then asks for a certain range of the resource

   • GET https://server.com/yadvs/api/v1/polls/123/votes/chart HTTP/1.1
     
     Range: bytes=0-1023
     

4. and the server answers

   • HTTP/1.1 206 Partial Content
     
     Accept-Ranges: bytes
     Content-Type: image/jpeg
     Content-Length: 1024
     Content-Range: bytes 0-1023/10922
     
     [...]
     

5. subsequently the client and server can continue the dialog until the entire resource is transferred.

Caching

ETags (Entity Tags) are HTTP headers that uniquely identify a resource and its version. They are used to implement efficient caching mechanisms in REST APIs. When a client requests a resource, the server sends an ETag along with the response. If the client makes subsequent requests for the same resource, it can include the previously received ETag in the If-None-Match header. If the resource hasn’t changed since the ETag was generated, the server returns a 304 Not Modified status code, indicating that the client can use its cached copy. This helps reduce network traffic and improve performance.

HTTP headers used in caching control:

• Age – contains the time in seconds the object was in a proxy cache.
• Cache-Control – holds directives (instructions) — in both requests and responses — that control caching in browsers and shared caches (e.g. Proxies, CDNs).
• Expires – contains the date/time after which the response is considered expired.
• ETag – is an identifier for a specific version of a resource. It lets caches be more efficient and save bandwidth, as a web server does not need to resend a full response if the content was not changed. Additionally, etags help to prevent simultaneous updates of a resource from overwriting each other (“mid-air collisions”).
• If-Modified-Since – makes the request conditional: the server sends back the requested resource, with a 200 status, only if it has been last modified after the given date. If the resource has not been modified since, the response is a 304 without any body; the Last-Modified response header of a previous request contains the date of last modification. Unlike If-Unmodified-Since, If-Modified-Since can only be used with a GET or HEAD.
• If-None-Match – makes the request conditional. For GET and HEAD methods, the server will return the requested resource, with a 200 status, only if it doesn’t have an ETag matching the given ones. For other methods, the request will be processed only if the eventually existing resource’s ETag doesn’t match any of the values listed.
• Last-Modified – contains a date and time when the origin server believes the resource was last modified. It is used as a validator to determine if the resource is the same as the previously stored one. Less accurate than an ETag header, it is a fallback mechanism. Conditional requests containing If-Modified-Since or If-Unmodified-Since headers make use of this field.
• Vary – describes the parts of the request message aside from the method and URL that influenced the content of the response it occurs in. Most often, this is used to create a cache key when content negotiation is in use.

Logging and Monitoring

Application servers will provide means to intercept and log every REST request and response.

The idea is to implement some sort of request context for each individual API call. Tag each call with and UUID and retain a timestamp. Use this tag when logging so that it will be easy to identify each particular trace in the log repository. Particularly in case of an error display the tag and the timestamp to simplify the analysis of the issue.

Using Messages and Error Handling Consistently

Use JSON for request and response bodies and in order to do that use the Accept and Content-Type Headers.

If there are fields that appear often in requests and responses they should be documented [Google Cloud].

In case of an error response code provide a body that will help identify and resolve the issue, e.g.:

{
    "errors": [
      {
        "code": "missing_field",
        "message": "must not be empty",
        "description": "Encountered during form validation",
        "info": "https://docs.mysite.com/user-registration#validation",
        "target": {
          "type": "field",
          "name": "email"
        }
      },
      {
        "code": "invalid_value",
        "message": "must not contain numbers",
        "description": "Encountered during form validation",
        "info": "https://docs.mysite.com/user-registration#validation",
        "target": {
          "type": "field",
          "name": "name"
        }
      },
      {
        "code": "invalid_value",
        "message": "numeric value out of bounds (<12 digits&>.<0 digits> expected)",
        "description": "Encountered during form validation",
        "info": "https://docs.mysite.com/user-registration#validation",
        "target": {
          "type": "field",
          "name": "phoneNumber"
        }
      }
    ],
    "trace": "8773ec8c-e61b-4290-9d85-d672b9d34a59",
    "timestamp": "2024-10-07 16:28:34"
  }

Here the errors field contains an array of error conditions with detailed information while the trace and the timestamp fields point to the unique request ID that helps identifying the logging context.

To choose a project specific error format there are a number of options to use as a model: [Google Cloud], [IBM Cloud], [vnd.error], Problem Details for HTTP APIs [RFC 7807, RFC 9457], etc.

Documenting and Discovering the API

Open API

Open API (formerly Swagger) is a specification for defining REST APIs. It provides a formal standard for describing HTTP API endpoints, parameters, data models, and operations. This makes it easier for developers to understand, consume, and test APIs.

Spring Boot integrates seamlessly with Open API. By using annotations or configuration, one can generate an Open API specification for a Spring Boot application. This specification can then be used to automatically generate documentation, client code, and server stubs, simplifying API development and consumption.

HATEOAS

HATEOAS (Hypermedia As The Engine Of Application State) is a REST API design principle that emphasizes the use of hypermedia links to guide clients through an application. This means that instead of hardcoding URLs, clients follow links provided in the API responses to discover available actions and resources.

HATEOAS promotes loose coupling between clients and servers, making APIs more adaptable and resilient to change. It also encourages the creation of self-documenting APIs, as the links and their associated metadata provide information about how to interact with the system.

Most popular hypertext link specs [Building Your API for Longevity]:

• HAL – JSON Hypertext Application Language (multiple versions)
• JSON-LD – A JSON-based Serialization for Linked Data
• JSON API – A Specification for Building APIs in JSON
• Collection+JSON – Hypermedia Type
• Siren – a hypermedia specification for representing entities
• CPHL – Cross-Platform Hypertext Language

References

1. Google Cloud – API design guide
2. IBM Cloud – API Handbook
3. Mathieu Fenniak – The Web API Checklist
4. Microsoft Azure – RESTful web API design
5. Mozilla Developer Network (MDN) – HTTP request methods
6. NGINX Community Blog – Building Your API for Longevity – Part 1 (Spec-Driven Development) and Part 2 (Best Practices)
7. OData – Open Data Protocol
8. Open API and springdoc-openapi