Wjjsoft Structured Storage Library (SSG-5)

Introduction

Wjjsoft Structured Storage Library provides a 64-bit cross-platform solution for applications to store information with hierarchical folders inside a single storage file. The information can be any type of files, such as text, webpages, images, documents, spreadsheets, presentations, videos and any other arbitrary files, no matter whether they're text or binary files, all information is automatically compressed and saved in hierarchical folders inside a single storage file.

The 64-bit storage library supports large files (Size>2GiB, up to 8EiB), Instant-load and Load-on-access features, and includes a shell command line tool, a recovery tool and a set of APIs that can be used to manipulate file/folder entries either from command line or programmatically.

Key Features

The latest version of the structured storage library is 5.0. Significant improvements have been made since the previous version 3.x, which was initially released in 2000.

Files and Folders (Tree structured)

Files and Folders are the basic concepts of the storage library. All information and data are stored with in inner files located in inner folders that are tree-structured. Each inner folder maintains a collection of inner files and a collection of sub folders. The inner files and folders are enumerable and searchable by names. The inner file/folder names conform to the common filename conventions, but without limits in length, therefore can be programmatically renamed with any long filenames.

Data Compression

All information is automatically compressed with Zlib to help save disk space (up to 90%) without too much loss of performance.

Data Integrity Checksum

For each inner files/folders, the data integrity checksum values are respectively saved with them in the storage, so the storage library is able to test data integrity while extracting information, and ensure that data is intact.

Large File Support

With 64-bit programming, large files are supported with no 2GiB limitations any more. In theory, capacity of a single storage can be extended up to 8EiB (2^63 or 10^18).

Load-on-Access

Only a small data file header is required to load when a storage initially opens, all additional files/folders are lazy-loaded on access.

Transaction Commit

The storage state is reverted to its last commited state when closing a storage without committing changes.

Storage Space Recycling

A recycle bin is internally maintained when allocating and releasing disk space inside a storage.

Storage Size Optimization

Releasing files/folders may temporarily produce fragments, that may be reallocated when adding additional files/folders, and the optimization procedure is also inbuilt for actively eliminating fragments to keep the storage compact.

Recoverability

In case of storage corrupted, the recovery tool is provided for rescuing data from in the storage if any inner filies/folders are still in good condition. The data checksum helps determine which inner files/folders are in good state and can fully be retrieved. The recovery tool skips the storage file header and looks into all existing data slices and seeks for any possibilities of retrieving individual files/folders that are still there intact.

In-memory Storage Support

All data interfaces are based on streams with random access features, so a storage can be created either over a disk file or in a memory buffer.

Cross-platform Support

The library was written in ANSI C/C++ with no platform specific features required, and works with those familiar operation systems (e.g. Linux/FreeBSD/macOS/Windows).

Benefits

Saving All Information in a Single File

It can be a great solution to store all information in the tree-structured form inside a single disk file, without having to make mess in the file system for storing a number of small (or large) files. For example, it may help a lot with management of a large number of software configurations, personal information and various types of documents.

Fully Customizable Folder Tree

Each storage internally maintains a fully customizable folder (directory) tree. Like the local file system, it allows applications to create any number of files, folders and sub folders inside the storage, without having to care about how the disk space is allocated.

A Great Solution to Store Unstructured Info

With the fully customizable hierarchical folders concept, it can be a great solution to store and organize a great deal of unstructured information, such as images, documents, spreadsheets, presentations, videos and any ordinary files. Since 1998, the Mybase software has been a such program that helps many users run their own knowledge base in the tree outline form effectively and efficiently.

Data Compression for Saving Disk Space

By utilizing the Zlib compression library, all information is automatically compressed before putting into the storage in order to save disk space.

Instant-Open and Load-on-Access

Opening a storage is very fast even for large-sized storage files. It usually takes less than one second to open a storage.

Storage Size Optimization

By running the size-optimization procedure, most of space fragments can be eliminated from the storage, that helps save disk space and save time when archiving or creating backup, or transmitting over network.

Recoverability from corrupted storage

The recovery tool is available for rescuing information in case of corrupted storage.

Supported Platforms

The following operating systems are supported.

  • Linux
  • FreeBSD
  • macOS
  • Windows

Shell Command Line

Currently the storage library includes a shell command line and a recovery tool built for Windows Console and Unix Shell.

SSG Shell

The shell command line program name is 'ssg', which accepts an operation tag with a few arguments followed, a typical command line may look like this: "./ssg new -stg=1.nyf -passwd=abc" where the 'new' is recognized as a sub command tag, with two arguments followed: '-stg=1.nyf -passwd=abc'; Some of commands allow arguments with no tags, that are usually required to run the commands. For example, the '-stg=' is a required parameter for the 'ssg new' command, so it can be omitted (well, must be typed in the original order), like this: "./ssg new 1.nyf -passwd=abc".

  1. ssg new: Create a storage over a specified disk file.
    • -stg=[file-path]: specifies a filename including path to create a storage in the file (Required*);
    • -version=[3 or 5]: determines which version of the storage library is selected; Default value: 5;
    • -zlib=[0-9]: determines the zlib compression level; 0: no compression, 9: maximum compression ratio; Default value: 9;
    • -passwd=[password]: Set a password for the newly created storage;
    • -offset=[#]: specifies a positive number of bytes for the storage to offset; Default value: 0;
    • -signature=[xxxx]: specifies a 4-byte application-defined signature (aka. magic number) which is saved within the storage header; Default value: ":NYF";
  2. ssg folder: Create a folder entry by a specified SSG path.
    • -stg=[file-path]: specifies a file path to the destination storage (Required*);
    • -path=[ssg-path]: specifies an inner SSG path to the folder entry being created (Required*);
    • -offset=[#]: specifies a positive number of bytes for the storage to offset; It must be the exact number which was input during the 'ssg new' command; Default value: 0;
  3. ssg file: Create a file entry without inserting any content.
    • -stg=[file-path]: specifies a file path to the destination storage (Required*);
    • -fn=[ssg-path]: specifies an inner SSG path to the desitnation file entry being created (Required*);
    • -offset=[#]: specifies a positive number of bytes for the storage to offset; It must be the exact number which was input during the 'ssg new' command; Default value: 0;
  4. ssg import: Create a file entry with content of the specified file.
    • -stg=[file-path]: specifies a file path to the destination storage (Required*);
    • -fndst=[ssg-path]: specifies an inner SSG path to the desitnation file entry being created (Required*);
    • -fnsrc=[file-path]: specifies an disk file path to the source content (Required*); If not supplied, it attempts to accept text input from console/keyboard (aka. stdin or std::cin) as the new file entry's content;
    • -offset=[#]: specifies a positive number of bytes for the storage to offset; It must be the exact number which was input during the 'ssg new' command; Default value: 0;
  5. ssg export: Export an inner file content to a specified disk file.
    • -stg=[file-path]: specifies a file path to the source storage (Required*);
    • -fnsrc=[ssg-path]: specifies an inner SSG path to the source file entry being exported (Required*);
    • -fndst=[file-path]: specifies an disk file to save content (Required*); If not supplied, it attempts to print content on console/screen (aka. stdout or std::cout);
    • -offset=[#]: specifies a positive number of bytes for the storage to offset; It must be the exact number which was input during the 'ssg new' command; Default value: 0;
  6. ssg hint: Set hint info for the specified file or folder entry. The 'hint' info can be a short text that describes the folder/file entry, but is different than the file/folder's name,
    • -stg=[file-path]: specifies a file path to the destination storage (Required*);
    • -entry=[ssg-path]: specifies an inner SSG path to the desitnation file/folder entry (Required*);
    • -hint=[text]: specifies a hint text for the file/folder entry;
    • -act=[get|set]: specifies an act for getting or setting hint text for the file/folder entry;
    • -offset=[#]: specifies a positive number of bytes for the storage to offset; It must be the exact number which was input during the 'ssg new' command; Default value: 0;
  7. ssg list: List out sub entries of a folder entry.
    • -stg=[file-path]: specifies a file path to the storage (Required*);
    • -path=[ssg-path]: specifies an inner SSG path to the folder entry being listed (Required*); Default value: /;
    • -type=[files|folders|all]: specifies what type of entries to be listed; Default value: all;
    • -fields=[name|date|time|hint|size/packed-size/extent-size|all]: specifies which fields of the entries to be displayed; Default value: all+size;
    • -offset=[#]: specifies a positive number of bytes for the storage to offset; It must be the exact number which was input during the 'ssg new' command; Default value: 0;
  8. ssg delete: Delete an inner file/folder entry.
    • -stg=[file-path]: specifies a file path to the destination storage (Required*);
    • -entry=[ssg-path]: specifies an inner SSG path to the file/folder entry being deleted (Required*);
    • -offset=[#]: specifies a positive number of bytes for the storage to offset; It must be the exact number which was input during the 'ssg new' command; Default value: 0;
  9. ssg copy: Copy a folder tree branch from one storage to another.
    • -stgsrc=[file-path]: specifies a file path to the source storage (Required*);
    • -stgdst=[file-path]: specifies a file path to the destination storage (Required*);
    • -pathsrc=[ssg-path]: specifies an inner SSG path to the source folder entry being copied (Required*);
    • -pathdst=[ssg-path]: specifies an inner SSG path to the destination folder entry to accept content (Required*);
    • -offsrc=[#]: specifies a positive number of bytes for the source storage to offset; It must be the exact number which was input during the 'new' command; Default value: 0;
    • -offdst=[#]: specifies a positive number of bytes for the destination storage to offset; It must be the exact number which was input during the 'new' command; Default value: 0;
  10. ssg stat: Display a list of statistics info about the specified storage.
    • -stg=[file-path]: specifies a file path to a storage (Required*);
    • -offset=[#]: specifies a positive number of bytes for the storage to offset; It must be the exact number which was input during the 'new' command; Default value: 0;
  11. ssg digest: Calculate the digest value of the specified text info or file content.
    • -alg=[adler32|sha1|sha224|sha256|sha384|sha512]: specifies the digest algorithm (Required*);
    • -info=[text info]: specifies the text info to be calculated with the digest algorithm; The text info would need quotation marks if it contains blank spaces.
    • -fn=[file-path]: specifies a file path to an ordinary file to calculate checksum values;
  12. ssg copyfile: Copy content (or a portion) of an ordinary file to another one.
    • -fnsrc=[file-path]: specifies a file path to the source file being copied (Required*);
    • -fndst=[file-path]: specifies a file path to the destination file (Required*);
    • -offset=[#]: specifies a positive number of bytes as the starting point to copy;
    • -length=[#]: specifies a positive number of bytes to be copied to the destination file;

Recovery Tool

Using the recovery tool from in a Terminal window is straightforward, first change to the directory where the ssg5recover resides and then type a command line like this:

% cd path-to-ssg5recover % ./ssg5recover /pathTo/corrupted.nyf

The recovery tool attempts to seek as many possibilities as it can to rescue data in the storage. If any file/folder entries are successfully retrieved, a new storage file will be automatically generated in the same folder to keep the retrieved entries.

The recovery tool accepts a few more arguments, that may help in some cases retrieve more file/folder entries from in a given corrupted storage file or only data slices rescued from damaged disks.

  • -src=[file-path]: specifies a file path to a source storage file or data slice (Required*);
  • -dst=[file-path]: specifies a file path to the destination storage file;
  • -offset=[#]: specifies a positive number in bytes for the storage to offset within the source data slice file, if in the case it is only a part of the original storage file, for instance, it was rescued from a damaged disk;
  • -minsize=[#]: specifies a positive number in bytes, that you estimated on the minimal size of the original storage file;
  • -maxsize=[#]: specifies a positive number in bytes, that you estimated on the maximal size of the original storage file;

Downloads

To download the SSG-5 command line tools (including the recovery tool), please click the link below;

SSG-5 command line (.zip) for Linux/FreeBSD/macOS/Windows