Open Caching Logging Specification
Project Status:
(roll over for info)
Reviewing
The working group is actively reviewing the project output. This could be a document, software code, or data from a lab test.
Start:
April 1, 2022
Estimated Completion:
June 30, 2023
Problem Statement
This specification provides a framework for communication of CDN transactional logs between Open Caching participants to satisfy billing and operational requirements.
Project Description
CDNI Logging [RFC7937] defines requirements for exchange of log data between CDN participants including a Logging Reference Model, a Logging File Structure, and Logging Records along with an Atom-based index for retrieving the generated log files. Open Caching expands on the functionality defined in CDNI Logging to provide clear definitions and an expanded set of log record fields, additional mechanisms for transporting logs, new log container formats, log archive files, logging metadata sidecar files, new log record formats, configured inclusion of optional request and response header fields, and methods for advertising and configuring transformations of individual log fields.
In the future, this specification will be expanded further to cover other OCS component logs (e.g. Request Router), custom log container formats and log record formats as well as uCDN controlled configuration for aggregation, filtering, and sampling of log records.
Project Type
Document
Project Leads
Advisors
There are no SMEs associated with this project.
Goals and Objectives
- Provide a framework for communication of CDN request logs between Open Caching participants to satisfy billing and operational requirements.
- Clarify and extend missing functionality from the CDNI Logging Interface defined in RFC 7937.
- Allow batch push and streaming modes of operation in addition to the pull mode described in CDNI.
- Provide useful default and minimal configurations for the most common use cases, but allow custom configuration when the pre-defined configurations are not suitable.
Project Scope
This specification covers the following:
- Extensions to the pull mode of operation described in RFC7937
- Log container/file formats
- ELF
- JSON
- Newline delimited records (\n, \n\r)
- Archive containers (tar) containing other log files
- Log record fields
- Standard log record types
- A standard log record type is a combination of a set of fields in a defined textual format (JSON, CSV, or whitespace delimited).
- A ‘minimal’ type optimized for data size, including only the fields necessary to perform billing functions.
- A ‘standard’ type with a set of fields likely to be useful for operational monitoring.
- An ‘extended’ type with an exhaustive field set likely to cover most operational and troubleshooting use cases.
- Field transformations to support obfuscation, truncation, hashing, and encryption operations
- Optional custom request and response header field inclusion
- Logging transports supporting both push and pull modes of operation
- S3
- SFTP
- Kafka
- Generic custom transport
- Aggregation
- Filtering
- Auth
- Custom container formats
- Custom log record formats
- Custom field definitions
Contributors
The following members have contributed to this project. Click on their name to visit their profile. If they have not published their profile, the link will redirect to their LinkedIn profile.
Additional References
- [OC-LIFS] SVTA Open Caching Logging Integration Functional Specification (2017)
- [RFC7937] Content Distribution Network Interconnection (CDNI) Logging Interface
- [RFC8006] Content Delivery Network Interconnection (CDNI) Metadata
- [RFC8008] Content Delivery Network Interconnection (CDNI) Request Routing: Footprint and Capabilities Semantics
- [RFC4287] The Atom Syndication Format