Skip to content

Latest commit

 

History

History
221 lines (183 loc) · 7.89 KB

tip-298.md

File metadata and controls

221 lines (183 loc) · 7.89 KB
tip: 298
title: Reformat manifest	
author: [email protected]
discussions to: https://github.com/tronprotocol/TIPs/issues/298
status: DRAFT
type: Standards Track
category: Core
created: 2021-07-16

Simple Summary

This TIP describes the levelDB Manifest file reformatting function.

Abstract

The c++ version of levelDB has been running for a long time, the manifest file may grow too big, resulting in high memory and long start-up time when restarting the database, for this situation, use the java version to process the levelDB manifest file for regularization before starting the database.

Motivation

Optimize c++ version of levelDB startup, reduce memory usage and time.

Specification

  1. Create AccountAssetStore 2.Populate data when querying assets 3.Splitting when saving an account

1.Create AccountAssetStore Create a database to store split assets and save asset data.

Data structure of the AccountAsset

/* AccountAsset */
message AccountAsset {

  /* frozen balance */
  message Frozen {
    int64 frozen_balance = 1; // the frozen trx balance
    int64 expire_time = 2; // the expire time
  }

  // the create address
  bytes address = 1;

  // the other asset owned by this account
  map<string, int64> asset = 2;

  // the other asset owned by this account，key is assetId
  map<string, int64> assetV2 = 3;

  bytes asset_issued_name = 4;
  bytes asset_issued_ID = 5;

  map<string, int64> latest_asset_operation_time = 6;
  map<string, int64> latest_asset_operation_timeV2 = 7;

  map<string, int64> free_asset_net_usage = 8;
  map<string, int64> free_asset_net_usageV2 = 9;

  // frozen asset(for asset issuer)
  repeated Frozen frozen_supply = 10;
}

2.Populate data when querying assets When the data is split, only the account data is returned when the account is queried to reduce the size. If the assets in the account are used, the asset data is populated by querying the assets, and this operation is done automatically.

3.When saving the account, splitting is performed. When saving an account, the AccountStore.put method is used as a proxy for the AccountAssetStore.put method to store the account separately.

Implementation

The c++ version of levelDB is running, with the Compaction process, files are added and deleted, the levelDB sst file keeps changing, the manifest will keep growing (leveldb only rewrites this file when opening the database, it will only append at runtime), when it grows to a certain size, the database will restart, it may encounter OOM and cannot open the database.

  1. Check manifest size.
  2. Bulk loading of manifest using java version.
  3. reopen with leveldbjni

1.Check manifest size

// Read "CURRENT" file, which contains a pointer to the current manifest file
    File currentFile = new File(dir, Filename.currentFileName());
    if (!currentFile.exists()) {
      return false;
    }
    String currentName = com.google.common.io.Files.asCharSource(currentFile, UTF_8).read();
    if (currentName.isEmpty() || currentName.charAt(currentName.length() - 1) != '\n') {
      return false;
    }
    currentName = currentName.substring(0, currentName.length() - 1);
    File current = new File(dir, currentName);
    if (!current.isFile()) {
      return false;
    }
    long maxSize = options.maxManifestSize();
    if (maxSize < 0) {
      return false;
    }
    logger.info("CurrentName {}/{},size {} kb.", dir, currentName, current.length() / 1024);
    return current.length() >= maxSize * 1024 * 1024;

2.Bulk loading of manifest using java version.

        // open file channel
        try (FileInputStream fis = new FileInputStream(new File(databaseDir, currentName));
             FileChannel fileChannel = fis.getChannel()) {
            // read log edit log
            Long nextFileNumber = null;
            Long lastSequence = null;
            Long logNumber = null;
            Long prevLogNumber = null;
            Builder builder = new Builder(this, current);

            LogReader reader = new LogReader(fileChannel, throwExceptionMonitor(), true, 0);
            VersionEdit edit;
            for (Slice record = reader.readRecord(); record != null; record = reader.readRecord()) {
                // read version edit
                edit = new VersionEdit(record);

                // verify comparator
                // todo implement user comparator
                String editComparator = edit.getComparatorName();
                String userComparator = internalKeyComparator.name();
                checkArgument(editComparator == null || editComparator.equals(userComparator),
                    "Expected user comparator %s to match existing database comparator ", userComparator, editComparator);

                // apply edit
                builder.apply(edit);
                if (builder.batchSize > options.maxBatchSize()) {
                    Version version = new Version(this);
                    builder.saveTo(version);

                    // Install recovered version
                    finalizeVersion(version);

                    appendVersion(version);
                    builder = new Builder(this, current);
                }
                // save edit values for verification below
                logNumber = coalesce(edit.getLogNumber(), logNumber);
                prevLogNumber = coalesce(edit.getPreviousLogNumber(), prevLogNumber);
                nextFileNumber = coalesce(edit.getNextFileNumber(), nextFileNumber);
                lastSequence = coalesce(edit.getLastSequenceNumber(), lastSequence);
            }

/**
 * Apply the specified edit to the current state.
 */
public void apply(VersionEdit edit)
    {
    // Update compaction pointers
    for (Entry<Integer, InternalKey> entry : edit.getCompactPointers().entrySet()) {
    Integer level = entry.getKey();
    InternalKey internalKey = entry.getValue();
    versionSet.compactPointers.put(level, internalKey);
    }

    // Delete files
    for (Entry<Integer, Long> entry : edit.getDeletedFiles().entries()) {
    Integer level = entry.getKey();
    Long fileNumber = entry.getValue();
    if (!versionSet.options.fast()) {
    levels.get(level).deletedFiles.add(fileNumber);
    batchSize++;
    }

    // todo missing update to addedFiles?
    // missing update to addedFiles for open db to release resource
    if (levels.get(level).addedFilesMap.remove(fileNumber) != null) {
    batchSize--;
    }
    }

    // Add new files
    for (Entry<Integer, FileMetaData> entry : edit.getNewFiles().entries()) {
    Integer level = entry.getKey();
    FileMetaData fileMetaData = entry.getValue();

    // We arrange to automatically compact this file after
    // a certain number of seeks.  Let's assume:
    //   (1) One seek costs 10ms
    //   (2) Writing or reading 1MB costs 10ms (100MB/s)
    //   (3) A compaction of 1MB does 25MB of IO:
    //         1MB read from this level
    //         10-12MB read from next level (boundaries may be misaligned)
    //         10-12MB written to next level
    // This implies that 25 seeks cost the same as the compaction
    // of 1MB of data.  I.e., one seek costs approximately the
    // same as the compaction of 40KB of data.  We are a little
    // conservative and allow approximately one seek for every 16KB
    // of data before triggering a compaction.
    int allowedSeeks = (int) (fileMetaData.getFileSize() / 16384);
    if (allowedSeeks < 100) {
    allowedSeeks = 100;
    }
    fileMetaData.setAllowedSeeks(allowedSeeks);

    if (levels.get(level).deletedFiles.remove(fileMetaData.getNumber())) {
    batchSize--;
    }
    //levels.get(level).addedFiles.add(fileMetaData);
    levels.get(level).addedFilesMap.put(fileMetaData.getNumber(), fileMetaData);
    batchSize++;
    }
    }

3.When manifest is reach certain size ,open db with Bulk.

  public void open() throws IOException {
    DB database = factory.open(this.srcDbPath.toFile(), this.options);
    database.close();
    }

Rationale

Huge manifest costs a lot.