Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for compressions and make LZ4 default for backup 2.0 #833

Open
wants to merge 5 commits into
base: 3.x
Choose a base branch
from
Open

Add support for compressions and make LZ4 default for backup 2.0 #833

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a551e7e
initial change - TTL
arunagrawal-84 May 6, 2019
ff65515
Removing empty DSE extension module
arunagrawal-84 May 6, 2019
a1082b4
Change to instant for object lock
arunagrawal-84 Jun 7, 2019
404714c
add lz4 and no compression type
arunagrawal-84 Jul 30, 2019
645c058
Pass directives to backup file system.
arunagrawal-84 Sep 20, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions priam/src/main/java/com/netflix/priam/backup/AbstractFileSystem.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,13 @@
import com.netflix.priam.notification.EventObserver;
import com.netflix.priam.scheduler.BlockingSubmitThreadPoolExecutor;
import com.netflix.priam.utils.BoundedExponentialRetryCallable;
import com.netflix.priam.utils.DateUtil;
import com.netflix.spectator.api.patterns.PolledMeter;
import java.io.File;
import java.io.FileNotFoundException;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.time.Instant;
import java.util.Date;
import java.util.Iterator;
import java.util.List;
Expand Down Expand Up @@ -66,7 +68,7 @@ public abstract class AbstractFileSystem implements IBackupFileSystem, EventGene

// This is going to be a write-thru cache containing the most frequently used items from remote
// file system. This is to ensure that we don't make too many API calls to remote file system.
private final Cache<Path, Boolean> objectCache;
private final Cache<Path, Instant> objectCache;

@Inject
public AbstractFileSystem(
Expand Down Expand Up @@ -245,16 +247,16 @@ public Long retriableCall() throws Exception {
}

private void addObjectCache(Path remotePath) {
objectCache.put(remotePath, Boolean.TRUE);
objectCache.put(remotePath, DateUtil.getInstant());
}

@Override
public boolean checkObjectExists(Path remotePath) {
// Check in cache, if remote file exists.
Boolean cacheResult = objectCache.getIfPresent(remotePath);
Instant lastUpdatedTimestamp = objectCache.getIfPresent(remotePath);

// Cache hit. Return the value.
if (cacheResult != null) return cacheResult;
if (lastUpdatedTimestamp != null) return true;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we do not really check the value of the timestamp, what difference does this make relative to having a Boolean? Maybe I will discover its importance in other fails, if I do, will delete this comment.


// Cache miss - Check remote file system if object exist.
boolean remoteFileExist = doesRemoteFileExist(remotePath);
Expand Down