jlengrand/RemarkablePocket
1
0
Fork 0
mirror of https://github.com/jlengrand/RemarkablePocket.git synced 2026-03-10 08:41:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #7 from nov1n/feature/sync15support

Browse Source 
Add support for new sync protocol
This commit is contained in:
Robert Carosi
2022-09-05 10:58:02 +02:00
committed by GitHub
parent deb4d96bce a91ca9e914
commit 33711aa8f5
21 changed files with 440 additions and 370 deletions
Download Patch File Download Diff File Expand all files Collapse all files

79
README.md
View File

@@ -1,6 +1,8 @@
![Example article](assets/logo-title.png)
*Remarkable Pocket* synchronizes articles from [Pocket](https://getpocket.com) to your [Remarkable](https://remarkable.com/) tablet. It can be run on your computer or on a server. Because it does not run on the device itself this approach saves battery life, and is resistant to Remarkable software updates.
*Remarkable Pocket* synchronizes articles from [Pocket](https://getpocket.com) to
your [Remarkable](https://remarkable.com/) tablet. It can be run on your computer or on a server. Because it does not
run on the device itself this approach saves battery life, and is resistant to Remarkable software updates.
An example run of the program can be found below:
@@ -32,28 +34,48 @@ An example run of the program can be found below:
</details>
## Features
- **No installation required.** The application can be run with a single command.
- **Works on Remarkable 1 and 2.**
- **Works on Remarkable 1 and 2.**
- **Full support for images, code blocks, and formulas.**
- **Articles are downloaded as epubs.** This allows you to customize the font, font size, margins, etc.
- **Automatically archive read articles on Pocket.** When you finish reading an article and close it while on the last page, it will be automatically deleted from the Remarkable and archived on Pocket. A new unread article will be downloaded in its place.
- **Download articles from Pocket with a given tag.** If a `tag-filter` (see [Configuration](#configuration)) is supplied then only articles with that tag will be downloaded.
- **Automatically archive read articles on Pocket.** When you finish reading an article and close it while on the last
page, it will be automatically deleted from the Remarkable and archived on Pocket. A new unread article will be
downloaded in its place.
- **Download articles from Pocket with a given tag.** If a `tag-filter` (see [Configuration](#configuration)) is
supplied then only articles with that tag will be downloaded.
## Usage
The easiest way to run the application is using Docker. First install Docker for your platform from https://docs.docker.com/get-docker/. Then run the following command to start the application on Linux or Mac (I have not tested it on Windows yet):
The easiest way to run the application is using Docker. First install Docker for your platform
from https://docs.docker.com/get-docker/. Then run the following command to start the application on Linux or Mac (I
have not tested it on Windows yet):
```
touch ~/.remarkable-pocket && docker run -it --env TZ=$(date +%Z) -p 65112:65112 -v ~/.remarkable-pocket:/root/.remarkable-pocket ghcr.io/nov1n/remarkable-pocket:0.0.3
touch ~/.remarkable-pocket ~/.rmapi && mkdir -p ~/.rmapi-cache && docker run -it --env TZ=Europe/Amsterdam -p 65112:65112 -v ~/.remarkable-pocket:/root/.remarkable-pocket -v ~/.rmapi:/root/.rmapi -v ~/.rmapi-cache:/root/.cache/rmapi ghcr.io/nov1n/remarkable-pocket:0.2.0
```
The first time you run the application, you will be asked to authorize Pocket and Remarkable Cloud. Once you have done this subsequent runs will read the credentials from the `~/.remarkable-pocket` file.
By default articles are synchronized to the `/Pocket/` directory on the Remarkable every 30 minutes.
The first time you run the application, you will be asked to authorize Pocket and Remarkable Cloud. Once you have done
this subsequent runs will read the credentials from the `~/.remarkable-pocket` and `~/.rmapi` file. You can also change
the timezone in the command to match your location.
*TIP:* If you want to launch the program on startup and keep it running in the background you can use *launchd* (on Mac) or *systemd* (on Linux). On Mac copy [this](nl.carosi.remarkable-pocket.plist) file to `~/Library/LaunchAgents/` followed by: `launchctl load -w nl.carosi.remarkable-pocket.plist`. Logs will be sent to `~/.remarkable-pocket.log`.
By default, articles are synchronized to the `/Pocket/` directory on the Remarkable every 60 minutes.
*TIP:* If you want to launch the program on startup and keep it running in the background you can use *launchd* (on Mac)
or *systemd* (on Linux). On Mac copy [this](nl.carosi.remarkable-pocket.plist) file to `~/Library/LaunchAgents/`
followed by: `launchctl load -w nl.carosi.remarkable-pocket.plist`. Logs will be sent to `~/.remarkable-pocket.log`.
## Raspberry Pi
There is also a Docker image available for the Raspberry Pi, so the command in [Usage](#usage) will work. You do need a
browser to complete the authentication flow. If your Pi runs without a screen I recommend using a VNC client when
running the program for the first time.
## Configuration
The default configuration can be changed by providing command-line arguments. Simply append these to the `docker run` command. Below is a list of all available options.
The default configuration can be changed by providing command-line arguments. Simply append these to the `docker run`
command. Below is a list of all available options.
```
Usage: remarkable-pocket [-hnorV] [-d=<storageDir>] [-f=<tagFilter>] [-i=<interval>] [-l=<articleLimit>]
Synchronizes articles from Pocket to the Remarkable tablet.
@@ -66,7 +88,6 @@ Synchronizes articles from Pocket to the Remarkable tablet.
Default: 10
-i, --interval=<interval> The interval between subsequent synchronizations.
Default: 60m
-r, --reset-credentials Reset all credentials.
-d, --storage-dir=<storageDir>
The storage directory on the Remarkable in which to store downloaded Pocket articles.
Default: /Pocket/
@@ -77,25 +98,45 @@ Synchronizes articles from Pocket to the Remarkable tablet.
```
## Limitations
- Articles behind a paywall cannot be downloaded.
- Articles on websites with sophisticated DDOS protection cannot be downloaded.
- Articles that use javascript to load the the content cannot be downloaded.
- Articles on websites with DDOS protection or captcha cannot be downloaded.
- Articles that use javascript to load the content cannot be downloaded.
## Build Instructions
The project uses Gradle as a build tool and can be built using the `gradle build` command. This will generate jars in `build/libs/`. To run the jar, use the `java -jar build/libs/remarkable-pocket-x.x.x.jar` command, replacing `x.x.x` with the correct version.
The project uses Gradle as a build tool and can be built using the `gradle build` command. This will generate jars
in `build/libs/`. To run the jar, use the `java -jar build/libs/remarkable-pocket-x.x.x.jar` command, replacing `x.x.x`
with the correct version.
### Building docker
To build the docker image run `gradle jib`. This will use a dynamically generated Dockerfile based on the configuration in the `jib` section of the `build.gradle` file. To run the image, see the command in the [Usage](#usage) section.
To build the docker image run `gradle jib`. This will use a dynamically generated Dockerfile based on the configuration
in the `jib` section of the `build.gradle` file. To run the image, see the command in the [Usage](#usage) section.
### Other package formats
If you would like to package the application in another format e.g. `deb`, `nix` or `AUR`, I'm happy to review a Pull Request.
If you would like to package the application in another format e.g. `deb`, `nix` or `AUR`, I'm happy to review a Pull
Request.
### Disclaimer
RemarkablePocket uses rmapi to connect to Remarkable cloud. The newly released sync protocol is not yet tested through
and may contain bugs.
As [recommended](https://github.com/juruen/rmapi#warning-experimental-support-for-the-new-sync-protocol) by rmapi please
make sure you have a backup of your files.
## Thanks
- https://epub.press/ for providing a free epub generator API. Consider donating to support this project.
- https://github.com/jlarriba/jrmapi for providing a Java API for the Remarkable Cloud.
- https://github.com/juruen/rmapi for providing a client for the Remarkable Cloud.
## Support
[!["Buy Me A Coffee"](https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png)](https://www.buymeacoffee.com/nov1n) if you want to say thanks. :-)
[!["Buy Me A Coffee"](https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png)](https://www.buymeacoffee.com/nov1n)
if you want to say thanks. :-)
## Disclaimer
The author(s) and contributor(s) are not associated with reMarkable AS, Norway. reMarkable is a registered trademark of reMarkable AS in some countries. Please see https://remarkable.com for their product.
The author(s) and contributor(s) are not associated with reMarkable AS, Norway. reMarkable is a registered trademark of
reMarkable AS in some countries. Please see https://remarkable.com for their product.

27
build.gradle
View File

@@ -1,13 +1,13 @@
plugins {
id "org.springframework.boot" version "2.5.5"
id "io.spring.dependency-management" version "1.0.11.RELEASE"
id "com.google.cloud.tools.jib" version "3.1.4"
id "com.google.cloud.tools.jib" version "3.3.0"
id "com.github.johnrengelman.shadow" version "7.1.0"
id "java"
}
group = "nl.carosi"
version = "0.0.3"
version = "0.2.0"
java {
toolchain {
@@ -17,16 +17,32 @@ java {
jib {
from {
image = "azul/zulu-openjdk-alpine:17-jre-headless"
image = "eclipse-temurin:17-jre"
platforms {
platform {
architecture = 'amd64'
os = 'linux'
}
platform {
architecture = 'arm'
os = 'linux'
}
}
}
container {
jvmFlags = ["-Xshare:auto", "-XX:TieredStopAtLevel=1", "-XX:CICompilerCount=1", "-XX:+UseSerialGC", "-Xmx512m"]
format = "OCI"
}
to {
image = "ghcr.io/nov1n/remarkable-pocket"
tags = [project.version.toString()]
}
extraDirectories {
paths = 'src/main/jib'
permissions = [
'/usr/local/bin/rmapi*': '755'
]
}
}
repositories {
@@ -41,9 +57,6 @@ dependencies {
implementation("com.positiondev.epublib:epublib-core:3.1") {
exclude group: "org.slf4j"
}
implementation("es.jlarriba:jrmapi:0.7") {
exclude group: "org.apache.logging.log4j"
}
implementation "net.lingala.zip4j:zip4j:1.2.4"
implementation "info.picocli:picocli:4.6.1"
implementation "com.fasterxml.jackson.core:jackson-core:2.13.0"

12
docker-compose.yml Normal file
View File

@@ -0,0 +1,12 @@
version: '3'
services:
remarkable-pocket:
image: ghcr.io/nov1n/remarkable-pocket:0.2.0
restart: always
ports:
- 65112:65112
volumes:
- ~/.remarkable-pocket:/root/.remarkable-pocket
- ~/.rmapi:/root/.rmapi
- ~/.rmapi-cache:/root/.cache/rmapi

4
nl.carosi.remarkable-pocket.plist
View File

@@ -6,9 +6,9 @@
<string>nl.carosi.remarkable-pocket</string>
<key>ProgramArguments</key>
<array>
<string>/bin/sh</string>
<string>/bin/sh</string>
<string>-c</string>
<string>while ! /usr/local/bin/docker version > /dev/null 2>&amp;1; do sleep 5; done &amp;&amp; /usr/local/bin/docker run --env TZ=$(date +%Z) -v ~/.remarkable-pocket:/root/.remarkable-pocket ghcr.io/nov1n/remarkable-pocket:0.0.3 1>>$HOME/.remarkable-pocket.log 2>&amp;1</string>
<string>while ! /usr/local/bin/docker version > /dev/null 2>&amp;1; do sleep 5; done &amp;&amp; /usr/local/bin/docker run --env TZ=Europe/Amsterdam -v ~/.remarkable-pocket:/root/.remarkable-pocket -v ~/.rmapi:/root/.rmapi -v ~/.rmapi-cache:/root/.cache/rmapi ghcr.io/nov1n/remarkable-pocket:0.2.0 1>>$HOME/.remarkable-pocket.log 2>&amp;1</string>
</array>
<key>RunAtLoad</key>
<true/>

8
src/main/java/nl/carosi/remarkablepocket/ArticleDownloader.java
View File

@@ -67,7 +67,7 @@ class ArticleDownloader {
try {
return tryDownloadImpl(article, storageDir);
} catch (IOException | RuntimeException e) {
LOG.error("Failed to download article {}.", e.getMessage());
LOG.error("Failed to download article: {}.", e.getMessage());
LOG.debug("Stack trace: ", e);
return Optional.empty();
}
@@ -165,9 +165,9 @@ class ArticleDownloader {
}
}
private final record DownloadRequest(String author, String publisher, String[] urls) {}
private record DownloadRequest(String author, String publisher, String[] urls) {}
private final record DownloadResponse(String id) {}
private record DownloadResponse(String id) {}
private final record StatusResponse(String message, int progress) {}
private record StatusResponse(String message, int progress) {}
}

77
src/main/java/nl/carosi/remarkablepocket/AuthService.java
View File

@@ -1,77 +0,0 @@
package nl.carosi.remarkablepocket;
import static java.time.temporal.ChronoUnit.MINUTES;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import es.jlarriba.jrmapi.Jrmapi;
import java.lang.reflect.Field;
import java.time.Instant;
import java.time.ZoneId;
import java.time.format.DateTimeFormatter;
import java.time.format.FormatStyle;
import java.util.Base64;
import java.util.concurrent.atomic.AtomicReference;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
// Hacky, but unfortunately jrmapi does not refresh bearer tokens.
final class AuthService {
private static final Logger LOG = LoggerFactory.getLogger(AuthService.class);
private static final Base64.Decoder B64_DECODER = Base64.getDecoder();
private static final DateTimeFormatter TEMPORAL_FORMATTER =
DateTimeFormatter.ofLocalizedDateTime(FormatStyle.SHORT)
.withZone(ZoneId.systemDefault());
private final ObjectMapper objectMapper;
private final AtomicReference<Jrmapi> rmapi;
private final String deviceToken;
private Instant nextRefresh;
public AuthService(
ObjectMapper objectMapper,
AtomicReference<Jrmapi> rmapi,
@Value("${rm.device-token}") String deviceToken) {
this.objectMapper = objectMapper;
this.rmapi = rmapi;
this.deviceToken = deviceToken;
}
void ensureValid() {
if (nextRefresh != null && Instant.now().isBefore(nextRefresh)) {
return;
}
try {
refreshToken();
} catch (NoSuchFieldException | IllegalAccessException | JsonProcessingException e) {
LOG.debug("Exception occurred while refreshing auth token:", e);
throw new RuntimeException("Could not refresh auth token.");
}
}
private void refreshToken()
throws NoSuchFieldException, IllegalAccessException, JsonProcessingException {
Field tokenField = Jrmapi.class.getDeclaredField("userToken");
tokenField.setAccessible(true);
String token = (String) tokenField.get(rmapi.get());
Instant exp = getExpiration(token);
nextRefresh = exp.minus(30, MINUTES);
rmapi.set(new Jrmapi(deviceToken));
LOG.debug(
"Refreshed jrmapi token. Valid until: {}, refreshing at {}.",
TEMPORAL_FORMATTER.format(exp),
TEMPORAL_FORMATTER.format(nextRefresh));
}
private Instant getExpiration(String token) throws JsonProcessingException {
String[] chunks = token.split("\\.");
String payload = new String(B64_DECODER.decode(chunks[1]));
return objectMapper.readValue(payload, JWT.class).exp();
}
@JsonIgnoreProperties(ignoreUnknown = true)
private record JWT(Instant exp) {}
}

24
src/main/java/nl/carosi/remarkablepocket/MetadataProvider.java
View File

@@ -5,14 +5,10 @@ import static nl.carosi.remarkablepocket.ArticleDownloader.POCKET_ID_SEPARATOR;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.databind.ObjectMapper;
import es.jlarriba.jrmapi.Jrmapi;
import es.jlarriba.jrmapi.model.Document;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.concurrent.atomic.AtomicReference;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipInputStream;
@@ -22,7 +18,6 @@ import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathExpressionException;
import javax.xml.xpath.XPathFactory;
import nl.carosi.remarkablepocket.model.DocumentMetadata;
import nl.siegmann.epublib.epub.EpubReader;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.util.xml.SimpleNamespaceContext;
@@ -32,20 +27,15 @@ import org.xml.sax.SAXException;
final class MetadataProvider {
private static final Logger LOG = LoggerFactory.getLogger(MetadataProvider.class);
private final AtomicReference<Jrmapi> rmapi;
private final EpubReader epubReader;
private final RemarkableApi rmapi;
private final ObjectMapper objectMapper;
private final DocumentBuilder documentBuilder;
private final XPath publisherXpath;
private Path workDir;
public MetadataProvider(
AtomicReference<Jrmapi> rmapi,
EpubReader epubReader,
ObjectMapper objectMapper,
DocumentBuilder documentBuilder) {
RemarkableApi rmapi, ObjectMapper objectMapper, DocumentBuilder documentBuilder) {
this.rmapi = rmapi;
this.epubReader = epubReader;
this.objectMapper = objectMapper;
this.documentBuilder = documentBuilder;
this.publisherXpath = constructXpath();
@@ -66,11 +56,11 @@ final class MetadataProvider {
LOG.debug("Created temporary working directory: {}.", workDir);
}
DocumentMetadata getMetadata(Document doc) {
LOG.debug("Getting metadata for document: {}.", doc.getVissibleName());
DocumentMetadata getMetadata(String name) {
LOG.debug("Getting metadata for document: {}.", name);
ZipFile zip;
try {
zip = new ZipFile(rmapi.get().fetchZip(doc, workDir.toString() + File.separator));
zip = new ZipFile(rmapi.download(name, workDir.toString()));
} catch (IOException e) {
throw new RuntimeException(e);
}
@@ -81,7 +71,7 @@ final class MetadataProvider {
InputStream epub = zip.getInputStream(zip.getEntry(fileHash + ".epub"))) {
int pageCount = objectMapper.readValue(lines, Lines.class).pageCount();
String pocketId = extractPocketId(epub);
return new DocumentMetadata(doc, pageCount, pocketId);
return new DocumentMetadata(rmapi.info(name), pageCount, pocketId);
} catch (IOException | SAXException | XPathExpressionException e) {
throw new RuntimeException(e);
}
@@ -113,5 +103,5 @@ final class MetadataProvider {
}
@JsonIgnoreProperties(ignoreUnknown = true)
private final record Lines(int pageCount) {}
private record Lines(int pageCount) {}
}

122
src/main/java/nl/carosi/remarkablepocket/PocketAuthenticator.java Normal file
View File

@@ -0,0 +1,122 @@
package nl.carosi.remarkablepocket;
import static java.util.concurrent.TimeUnit.MINUTES;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Properties;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import pl.codeset.pocket.PocketAuth;
import pl.codeset.pocket.PocketAuthFactory;
public class PocketAuthenticator {
private static final Logger LOG = LoggerFactory.getLogger(PocketAuthenticator.class);
private static final String CONSUMER_KEY = "99428-51e4648a4528a1faa799c738";
private static final String TOKEN_PROPERTY = "pocket.access.token";
private final String authFile;
private final int port;
public PocketAuthenticator(
@Value("${pocket.auth.file}") String authFile,
@Value("${pocket.server.port}") int port) {
this.authFile = authFile;
this.port = port;
}
public PocketAuth getAuth() throws IOException {
Path authFilePath = Path.of(authFile);
Path authDirPath = authFilePath.getParent();
if (Files.notExists(authDirPath)) {
Files.createDirectories(authDirPath);
}
// TODO: Sometimes a directory is created, figure out why
if (Files.isDirectory(authFilePath)) {
Files.delete(authFilePath);
}
if (Files.exists(authFilePath) && Files.size(authFilePath) > 0) {
return authFromFile(authFilePath);
} else {
return authAndStore(authFilePath);
}
}
private PocketAuth authFromFile(Path authFilePath) throws IOException {
Properties properties = new Properties();
try (InputStream authStream = Files.newInputStream(authFilePath)) {
properties.load(authStream);
}
return PocketAuthFactory.createForAccessToken(
CONSUMER_KEY, properties.getProperty(TOKEN_PROPERTY));
}
private PocketAuth authAndStore(Path authFilePath) throws IOException {
PocketAuth auth = authenticate();
Properties properties = new Properties();
properties.setProperty(TOKEN_PROPERTY, auth.getAccessToken());
try (OutputStream authStream = Files.newOutputStream(authFilePath)) {
properties.store(authStream, null);
}
return auth;
}
private PocketAuth authenticate() throws IOException {
HttpServer server = HttpServer.create(new InetSocketAddress(port), 0);
ExecutorService execService = Executors.newSingleThreadExecutor();
server.setExecutor(execService);
server.createContext("/redirect", new RedirectHandler(server, execService));
server.start();
PocketAuthFactory factory =
PocketAuthFactory.create(CONSUMER_KEY, "http://localhost:" + port + "/redirect");
String authUrl = factory.getAuthUrl();
LOG.info("Visit {} and authorize this application.\n", authUrl);
try {
boolean terminated = execService.awaitTermination(5, MINUTES);
if (!terminated) {
throw new InterruptedException();
}
} catch (InterruptedException e) {
LOG.info("Pocket authorization timed out. Please try again.");
// System.exit doesn't work here. I suspect there is a deadlock in the
// 'logStream' method where it blocks on stdin.
Runtime.getRuntime().halt(1);
}
return factory.create();
}
private static final class RedirectHandler implements HttpHandler {
private final HttpServer server;
private final ExecutorService execService;
RedirectHandler(HttpServer server, ExecutorService execService) {
this.server = server;
this.execService = execService;
}
@Override
public void handle(HttpExchange exchange) throws IOException {
OutputStream outputStream = exchange.getResponseBody();
String res = "Authorization completed!";
exchange.sendResponseHeaders(200, res.length());
outputStream.write(res.getBytes());
outputStream.flush();
outputStream.close();
server.stop(0);
execService.shutdown();
}
}
}

40
src/main/java/nl/carosi/remarkablepocket/PocketService.java
View File

@@ -2,16 +2,15 @@ package nl.carosi.remarkablepocket;
import java.io.IOException;
import java.util.List;
import java.util.function.Consumer;
import java.util.stream.Collectors;
import javax.annotation.PostConstruct;
import nl.carosi.remarkablepocket.model.Article;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import pl.codeset.pocket.Pocket;
import pl.codeset.pocket.PocketAuth;
import pl.codeset.pocket.PocketAuthFactory;
import pl.codeset.pocket.modify.ArchiveAction;
import pl.codeset.pocket.modify.ModifyItemCmd;
import pl.codeset.pocket.modify.ModifyResult;
import pl.codeset.pocket.read.ContentType;
import pl.codeset.pocket.read.DetailType;
import pl.codeset.pocket.read.GetItemsCmd;
@@ -20,30 +19,14 @@ import pl.codeset.pocket.read.PocketItem;
import pl.codeset.pocket.read.Sort;
final class PocketService {
static final String CONSUMER_KEY = "99428-51e4648a4528a1faa799c738";
private static final Logger LOG = LoggerFactory.getLogger(PocketService.class);
private final String accessToken;
private final String tagFilter;
private Pocket pocket;
private final Pocket pocket;
public PocketService(
@Value("${pocket.access-token}") String accessToken,
@Value("${pocket.tag-filter}") String tagFilter) {
this.accessToken = accessToken;
public PocketService(@Value("${pocket.tag-filter}") String tagFilter, Pocket pocket) {
this.tagFilter = tagFilter;
}
static String getAccessToken(String redirectUrl, Consumer<String> prompt) throws IOException {
PocketAuthFactory factory = PocketAuthFactory.create(CONSUMER_KEY, redirectUrl);
prompt.accept(factory.getAuthUrl());
PocketAuth pocketAuth = factory.create();
return pocketAuth.getAccessToken();
}
@PostConstruct
private void auth() {
PocketAuth pocketAuth = PocketAuthFactory.createForAccessToken(CONSUMER_KEY, accessToken);
this.pocket = new Pocket(pocketAuth);
this.pocket = pocket;
}
List<Article> getArticles() throws IOException {
@@ -62,6 +45,13 @@ final class PocketService {
}
void archive(String id) throws IOException {
pocket.modify(new ModifyItemCmd.Builder().action(new ArchiveAction(id)).build());
ModifyResult res =
pocket.modify(new ModifyItemCmd.Builder().action(new ArchiveAction(id)).build());
if (res.getStatus() == 0) {
LOG.error(
"Could not archive article on Pocket: {}. Please archive it manually: https://getpocket.com/read/{}.",
res.getActionResults(),
id);
}
}
}

149
src/main/java/nl/carosi/remarkablepocket/RemarkableApi.java Normal file
View File

@@ -0,0 +1,149 @@
package nl.carosi.remarkablepocket;
import static java.lang.ProcessBuilder.Redirect.INHERIT;
import static java.nio.charset.StandardCharsets.UTF_8;
import static java.util.function.Predicate.not;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.util.Arrays;
import java.util.List;
import java.util.Scanner;
import java.util.function.Consumer;
import java.util.regex.Pattern;
import javax.annotation.PostConstruct;
import nl.carosi.remarkablepocket.model.Document;
import org.apache.logging.log4j.util.Strings;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.DependsOn;
@DependsOn("pocket") // Forces pocket auth to happen before rm auth
public class RemarkableApi {
private static final Logger LOG = LoggerFactory.getLogger(RemarkableApi.class);
private static final List<String> RMAPI_WARNING_PREFIXES =
List.of(
"Refreshing tree...",
"WARNING!!!",
" Using the new 1.5 sync",
" Make sure you have a backup");
private static final String RMAPI_EXECUTABLE =
"/usr/local/bin/rmapi"
+ (new File("/.dockerenv").exists()
? ("_" + System.getProperty("os.arch"))
: "");
private final String rmStorageDir;
private final ObjectMapper objectMapper;
public RemarkableApi(
ObjectMapper objectMapper, @Value("${rm.storage-dir}") String rmStorageDir) {
this.objectMapper = objectMapper;
this.rmStorageDir = rmStorageDir;
}
private static List<String> exec(String... command) {
return exec(List.<String[]>of(command));
}
private static List<String> exec(List<String[]> commands) {
List<ProcessBuilder> builders =
commands.stream()
.map(ProcessBuilder::new)
.peek(builder -> LOG.debug("Executing command: {}", builder.command()))
.toList();
try {
List<Process> processes = ProcessBuilder.startPipeline(builders);
Process last = processes.get(processes.size() - 1);
last.errorReader(UTF_8)
.lines()
.filter(line -> RMAPI_WARNING_PREFIXES.stream().noneMatch(line::startsWith))
.forEach(LOG::error);
return last.inputReader(UTF_8).lines().peek(LOG::debug).toList();
} catch (IOException e) {
throw new RuntimeException(e);
}
}
private static void logStream(InputStream src, Consumer<String> consumer) {
new Thread(
() -> {
Scanner sc = new Scanner(src);
sc.useDelimiter(Pattern.compile("\\n|: "));
while (sc.hasNext()) {
String token = sc.next();
if (token.equals("Refreshing tree...")) {
consumer.accept("Refreshing cache... This may take a while.");
} else {
consumer.accept(token);
}
}
})
.start();
}
@PostConstruct
public void login() {
try {
Process proc =
new ProcessBuilder(RMAPI_EXECUTABLE, "version").redirectInput(INHERIT).start();
logStream(proc.getInputStream(), LOG::info);
logStream(proc.getErrorStream(), LOG::error);
int exitCode = proc.waitFor();
LOG.info("");
if (exitCode != 0) {
throw new RuntimeException("Could not authenticate to Remarkable API");
}
} catch (IOException | InterruptedException e) {
throw new RuntimeException("Could not authenticate to Remarkable API", e);
}
}
public String download(String name, String dest) {
exec(RMAPI_EXECUTABLE, "-ni", "get", rmStorageDir + name);
exec("mv", name + ".zip", dest);
return dest + File.separator + name + ".zip";
}
public List<String> list() {
return exec(
List.of(
new String[] {RMAPI_EXECUTABLE, "-ni", "ls", rmStorageDir},
new String[] {"grep", "^\\[f\\]"},
new String[] {"cut", "-b5-"}));
}
public Document info(String name) {
List<String> info =
exec(
List.of(
new String[] {RMAPI_EXECUTABLE, "-ni", "stat", rmStorageDir + name},
new String[] {"sed", "/{/,$!d"}));
try {
return objectMapper.readValue(Strings.join(info, '\n'), Document.class);
} catch (JsonProcessingException e) {
throw new RuntimeException("Error parsing Remarkable API response", e);
}
}
public void upload(String path) {
exec(RMAPI_EXECUTABLE, "-ni", "put", path, rmStorageDir);
}
public void delete(String name) {
exec(RMAPI_EXECUTABLE, "-ni", "rm", rmStorageDir + name);
}
public void createDir(String path) {
List<String> parts = Arrays.stream(path.split("/")).filter(not(String::isEmpty)).toList();
for (int i = 1; i <= parts.size(); i++) {
String subdir = String.join("/", parts.subList(0, i));
exec(RMAPI_EXECUTABLE, "-ni", "mkdir", subdir);
}
}
}

42
src/main/java/nl/carosi/remarkablepocket/RemarkableService.java
View File

@@ -2,12 +2,8 @@ package nl.carosi.remarkablepocket;
import static com.google.common.base.Preconditions.checkArgument;
import es.jlarriba.jrmapi.Jrmapi;
import es.jlarriba.jrmapi.model.Document;
import java.nio.file.Path;
import java.util.List;
import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Stream;
import javax.annotation.PostConstruct;
import nl.carosi.remarkablepocket.model.DocumentMetadata;
import org.slf4j.Logger;
@@ -16,13 +12,12 @@ import org.springframework.beans.factory.annotation.Value;
final class RemarkableService {
private static final Logger LOG = LoggerFactory.getLogger(RemarkableService.class);
private final AtomicReference<Jrmapi> rmapi;
private final RemarkableApi rmapi;
private final MetadataProvider metadataProvider;
private final String rmStorageDir;
private String rmStorageDirId = "";
public RemarkableService(
AtomicReference<Jrmapi> rmapi,
RemarkableApi rmapi,
MetadataProvider metadataProvider,
@Value("${rm.storage-dir}") String rmStorageDir) {
if (!rmStorageDir.endsWith("/")) {
@@ -37,29 +32,20 @@ final class RemarkableService {
}
@PostConstruct
void findParentId() {
String[] dirs = rmStorageDir.substring(1, rmStorageDir.length() - 1).split("/");
for (final String dir : dirs) {
rmStorageDirId =
rmapi.get().listDocs().stream()
.filter(e -> e.getParent().equals(rmStorageDirId))
.filter(e -> e.getVissibleName().equals(dir))
.findFirst()
.map(Document::getID)
.orElseGet(() -> rmapi.get().createDir(dir, rmStorageDirId));
}
void createStorageDir() {
rmapi.createDir(rmStorageDir);
}
List<String> listDocumentNames() {
return docsStream().map(Document::getVissibleName).toList();
return rmapi.list();
}
List<DocumentMetadata> listReadDocuments() {
return docsStream()
return rmapi.list().stream()
.map(metadataProvider::getMetadata)
.peek(this::logPages)
// Current page starts counting at 0.
.filter(e -> e.doc().getCurrentPage() + 1 == e.pageCount())
.filter(e -> e.doc().currentPage() + 1 == e.pageCount())
.toList();
}
@@ -73,23 +59,19 @@ final class RemarkableService {
}
}
void delete(Document doc) {
rmapi.get().deleteEntry(doc);
void delete(String name) {
rmapi.delete(name);
}
private void logPages(DocumentMetadata meta) {
LOG.debug(
"{}: Current page: {}, page count: {}.",
meta.doc().getVissibleName(),
meta.doc().getCurrentPage() + 1,
meta.doc().name(),
meta.doc().currentPage() + 1,
meta.pageCount() == 0 ? "unknown" : meta.pageCount());
}
private Stream<Document> docsStream() {
return rmapi.get().listDocs().stream().filter(e -> e.getParent().equals(rmStorageDirId));
}
private void upload(Path path) {
rmapi.get().uploadDoc(path.toFile(), rmStorageDirId);
rmapi.upload(path.toAbsolutePath().toString());
}
}

22
src/main/java/nl/carosi/remarkablepocket/SyncApplication.java
View File

@@ -1,42 +1,46 @@
package nl.carosi.remarkablepocket;
import es.jlarriba.jrmapi.Jrmapi;
import java.util.concurrent.atomic.AtomicReference;
import java.io.IOException;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import nl.siegmann.epublib.epub.EpubReader;
import nl.siegmann.epublib.epub.EpubWriter;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Import;
import org.springframework.retry.annotation.EnableRetry;
import org.springframework.scheduling.annotation.EnableScheduling;
import pl.codeset.pocket.Pocket;
@SpringBootApplication
@EnableScheduling
@EnableRetry
@Import({
ArticleDownloader.class,
AuthService.class,
DownloadService.class,
EpubReader.class,
EpubWriter.class,
MetadataProvider.class,
PocketService.class,
PocketAuthenticator.class,
RemarkableApi.class,
RemarkableService.class,
SyncService.class,
})
public class SyncApplication {
@Bean
AtomicReference<Jrmapi> jrmapi(@Value("${rm.device-token}") String deviceToken) {
return new AtomicReference<>(new Jrmapi(deviceToken));
}
@Bean
DocumentBuilder documentBuilder() throws ParserConfigurationException {
DocumentBuilderFactory builderFactory = DocumentBuilderFactory.newInstance();
return builderFactory.newDocumentBuilder();
}
@Bean
Pocket pocket(PocketAuthenticator authenticator) {
try {
return new Pocket(authenticator.getAuth());
} catch (IOException e) {
throw new RuntimeException("Could not authenticate with Pocket", e);
}
}
}

162
src/main/java/nl/carosi/remarkablepocket/SyncCommand.java
View File

@@ -1,33 +1,13 @@
package nl.carosi.remarkablepocket;
import static java.util.Map.entry;
import static java.util.UUID.randomUUID;
import static java.util.concurrent.TimeUnit.MINUTES;
import static nl.carosi.remarkablepocket.ConnectivityChecker.ensureConnected;
import static nl.carosi.remarkablepocket.PocketService.getAccessToken;
import static org.springframework.boot.Banner.Mode.OFF;
import static org.springframework.boot.WebApplicationType.NONE;
import static picocli.CommandLine.Help.Visibility.ALWAYS;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;
import es.jlarriba.jrmapi.Authentication;
import java.io.Console;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.PrintStream;
import java.net.InetSocketAddress;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Map;
import java.util.Properties;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.function.Consumer;
import org.springframework.boot.builder.SpringApplicationBuilder;
import picocli.CommandLine;
import picocli.CommandLine.Command;
@@ -39,7 +19,7 @@ import picocli.CommandLine.Option;
sortOptions = false,
usageHelpAutoWidth = true,
// TODO: Read from gradle.properties
version = "0.0.3",
version = "0.2.0",
mixinStandardHelpOptions = true)
class SyncCommand implements Callable<Integer> {
@Option(
@@ -77,12 +57,6 @@ class SyncCommand implements Callable<Integer> {
showDefaultValue = ALWAYS)
private String interval;
@Option(
names = {"-r", "--reset-credentials"},
description = "Reset all credentials.",
arity = "0")
private boolean resetCredentials;
@Option(
names = {"-d", "--storage-dir"},
description =
@@ -120,9 +94,12 @@ class SyncCommand implements Callable<Integer> {
}
@Override
public Integer call() throws IOException {
public Integer call() {
ensureConnected(System.err::println);
// Handle sigterm (^C)
Runtime.getRuntime().addShutdownHook(new Thread(() -> Runtime.getRuntime().halt(1)));
Map<String, Object> cliProperties =
Map.ofEntries(
entry("pocket.archive-read", Boolean.toString(!noArchive)),
@@ -131,8 +108,13 @@ class SyncCommand implements Callable<Integer> {
entry("sync.interval", "PT" + interval),
entry("sync.run-once", Boolean.toString(runOnce)),
entry("pocket.tag-filter", tagFilter),
entry("logging.level." + this.getClass().getPackageName(), verbose ? "DEBUG" : "INFO")
);
entry("pocket.server.port", port),
entry(
"pocket.auth.file",
authFile.replaceFirst("^~", System.getProperty("user.home"))),
entry(
"logging.level." + this.getClass().getPackageName(),
verbose ? "DEBUG" : "INFO"));
new SpringApplicationBuilder(SyncApplication.class)
.logStartupInfo(false)
@@ -140,128 +122,8 @@ class SyncCommand implements Callable<Integer> {
.bannerMode(OFF)
.web(NONE)
.properties(cliProperties)
.properties(authProperties())
.run();
return 0;
}
private Properties authProperties() throws IOException {
Path authFilePath = Path.of(authFile.replaceFirst("^~", System.getProperty("user.home")));
Path authDirPath = authFilePath.getParent();
if (Files.notExists(authDirPath)) {
Files.createDirectories(authDirPath);
}
if (Files.exists(authFilePath) && Files.size(authFilePath) > 0 && !resetCredentials) {
Properties properties = new Properties();
try (InputStream authStream = Files.newInputStream(authFilePath)) {
properties.load(authStream);
}
return properties;
} else {
return createAuthProperties(authFilePath);
}
}
private Properties createAuthProperties(Path authFilePath) throws IOException {
Console console = System.console();
if (console == null) {
System.err.println(
"No console found. If you're using Docker please add the '-it' flags.\n");
System.exit(1);
}
console.printf("""
Welcome to Remarkable Pocket!
Please follow the next steps to connect your Pocket and Remarkable accounts. You only need to do this once.
""");
PrintStream origOut = System.out;
System.setOut(debugFilteringStream());
String rmDeviceToken = obtainRmDeviceToken(console);
System.setOut(origOut);
String pocketAccessToken = obtainPocketAccessToken(console);
Properties properties = new Properties();
properties.setProperty("pocket.access-token", pocketAccessToken);
properties.setProperty("rm.device-token", rmDeviceToken);
try (OutputStream authStream = Files.newOutputStream(authFilePath)) {
properties.store(authStream, null);
}
return properties;
}
// Hack to filter and format third-party logging until Spring is initialized.
private PrintStream debugFilteringStream() {
return new PrintStream(System.out) {
@Override
public void write(byte[] buf) {
String msg = new String(buf, StandardCharsets.UTF_8);
if(!msg.contains("DEBUG")) {
super.print(msg.split("- ")[1]);
}
}
};
}
private String obtainRmDeviceToken(Console console) {
String rmDeviceToken = null;
while (rmDeviceToken == null) {
String rmCloudCode =
console.readLine(
"Paste the code from https://my.remarkable.com/device/desktop/connect: ");
console.printf("Verifying... ");
rmDeviceToken = new Authentication().registerDevice(rmCloudCode, randomUUID());
}
console.printf("Success!\n");
return rmDeviceToken;
}
private String obtainPocketAccessToken(Console console) throws IOException {
HttpServer server = HttpServer.create(new InetSocketAddress(port), 0);
ExecutorService execService = Executors.newSingleThreadExecutor();
server.setExecutor(execService);
server.createContext("/redirect", new RedirectHandler(server, execService));
server.start();
Consumer<String> waitForUserAuth =
authUrl -> {
console.printf("Now visit %s and authorize this application.\n\n", authUrl);
try {
boolean terminated = execService.awaitTermination(1, MINUTES);
if (!terminated) {
throw new InterruptedException();
}
} catch (InterruptedException e) {
console.printf("Pocket authorization timed out. Please try again.\n");
System.exit(1);
}
};
return getAccessToken("http://localhost:" + port + "/redirect", waitForUserAuth);
}
private static final class RedirectHandler implements HttpHandler {
private final HttpServer server;
private final ExecutorService execService;
RedirectHandler(HttpServer server, ExecutorService execService) {
this.server = server;
this.execService = execService;
}
@Override
public void handle(HttpExchange exchange) throws IOException {
OutputStream outputStream = exchange.getResponseBody();
String res = "Authorization completed!";
exchange.sendResponseHeaders(200, res.length());
outputStream.write(res.getBytes());
outputStream.flush();
outputStream.close();
server.stop(0);
execService.shutdown();
}
}
}

18
src/main/java/nl/carosi/remarkablepocket/SyncService.java
View File

@@ -24,7 +24,6 @@ final class SyncService {
private final PocketService pocketService;
private final DownloadService downloadService;
private final RemarkableService remarkableService;
private final AuthService authService;
private final ApplicationContext appContext;
private final int articleLimit;
private final boolean archiveRead;
@@ -35,7 +34,6 @@ final class SyncService {
PocketService pocketService,
DownloadService downloadService,
RemarkableService remarkableService,
AuthService authService,
ApplicationContext appContext,
@Value("${rm.article-limit}") int articleLimit,
@Value("${pocket.archive-read}") boolean archiveRead,
@@ -44,7 +42,6 @@ final class SyncService {
this.pocketService = pocketService;
this.downloadService = downloadService;
this.remarkableService = remarkableService;
this.authService = authService;
this.appContext = appContext;
this.articleLimit = articleLimit;
this.archiveRead = archiveRead;
@@ -63,7 +60,6 @@ final class SyncService {
@Scheduled(fixedDelayString = "${sync.interval}")
void sync() {
ensureConnected(LOG::error);
authService.ensureValid();
try {
syncImpl();
@@ -114,18 +110,10 @@ final class SyncService {
LOG.info("Found {} read article(s) on Remarkable.", nDocs);
for (int i = 0; i < nDocs; i++) {
DocumentMetadata doc = documents.get(i);
LOG.info(
"({}/{}) Marking '{}' as read on Pocket...",
i + 1,
nDocs,
doc.doc().getVissibleName());
LOG.info("({}/{}) Marking '{}' as read on Pocket...", i + 1, nDocs, doc.doc().name());
pocketService.archive(doc.pocketId());
LOG.info(
"({}/{}) Deleting '{}' from Remarkable...",
i + 1,
nDocs,
doc.doc().getVissibleName());
remarkableService.delete(doc.doc());
LOG.info("({}/{}) Deleting '{}' from Remarkable...", i + 1, nDocs, doc.doc().name());
remarkableService.delete(doc.doc().name());
}
}
}

2
src/main/java/nl/carosi/remarkablepocket/model/Article.java
View File

@@ -1,7 +1,7 @@
package nl.carosi.remarkablepocket.model;
public final record Article(String id, String url, String title) {
public record Article(String id, String url, String title) {
public static Article of(String id, String url, String title) {
return new Article(id, url, sanitize(title));
}

6
src/main/java/nl/carosi/remarkablepocket/model/Document.java Normal file
View File

@@ -0,0 +1,6 @@
package nl.carosi.remarkablepocket.model;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
@JsonIgnoreProperties(ignoreUnknown = true)
public record Document(@JsonProperty("CurrentPage") int currentPage, @JsonProperty("VissibleName") String name) {}

4
src/main/java/nl/carosi/remarkablepocket/model/DocumentMetadata.java
View File

@@ -1,5 +1,3 @@
package nl.carosi.remarkablepocket.model;
import es.jlarriba.jrmapi.model.Document;
public final record DocumentMetadata(Document doc, int pageCount, String pocketId) {}
public record DocumentMetadata(Document doc, int pageCount, String pocketId) {}

BIN
src/main/jib/usr/local/bin/rmapi_amd64 Executable file
View File

Binary file not shown.

BIN
src/main/jib/usr/local/bin/rmapi_arm Executable file
View File

Binary file not shown.

1
src/main/resources/application.properties
View File

@@ -1,2 +1,3 @@
logging.level.nl.siegmann.epublib=OFF
#logging.level.nl.carosi.remarkablepocket=DEBUG
logging.pattern.console=[%d{yyyy-MM-dd HH:mm:ss}] %clr(%m){faint}%n

11
src/test/java/nl/carosi/remarkablepocket/SyncCommandTests.java
View File

@@ -1,11 +0,0 @@
package nl.carosi.remarkablepocket;
import org.junit.jupiter.api.Test;
class SyncCommandTests {
@Test
void contextLoads() {}
@Test
void test() {}
}