Skip to content

Commit

Permalink
Merge branch 'master' into feat-itunes-2
Browse files Browse the repository at this point in the history
  • Loading branch information
@dvikan
dvikan authored Jan 9, 2024
2 parents 0dda1ab + ea58c8d commit 3733bc1
Show file tree
Hide file tree
Showing 67 changed files with 2,428 additions and 426 deletions.
2 changes: 1 addition & 1 deletion Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ ENV CURL_IMPERSONATE ff91esr

COPY ./config/nginx.conf /etc/nginx/sites-available/default
COPY ./config/php-fpm.conf /etc/php/8.2/fpm/pool.d/rss-bridge.conf
COPY ./config/php.ini /etc/php/8.2/fpm/conf.d/90-rss-bridge.conf
COPY ./config/php.ini /etc/php/8.2/fpm/conf.d/90-rss-bridge.ini

COPY --chown=www-data:www-data ./ /app/

Expand Down
164 changes: 138 additions & 26 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,18 @@

![RSS-Bridge](static/logo_600px.png)

RSS-Bridge is a web application.
RSS-Bridge is a PHP web application.

It generates web feeds for websites that don't have one.

Officially hosted instance: https://rss-bridge.org/bridge01/

IRC channel #rssbridge at https://libera.chat/


[![LICENSE](https://img.shields.io/badge/license-UNLICENSE-blue.svg)](UNLICENSE)
[![GitHub release](https://img.shields.io/github/release/rss-bridge/rss-bridge.svg?logo=github)](https://github.com/rss-bridge/rss-bridge/releases/latest)
[![irc.libera.chat](https://img.shields.io/badge/irc.libera.chat-%23rssbridge-blue.svg)](https://web.libera.chat/#rssbridge)
[![Chat on Matrix](https://matrix.to/img/matrix-badge.svg)](https://matrix.to/#/#rssbridge:libera.chat)
[![Actions Status](https://img.shields.io/github/actions/workflow/status/RSS-Bridge/rss-bridge/tests.yml?branch=master&label=GitHub%20Actions&logo=github)](https://github.com/RSS-Bridge/rss-bridge/actions)

|||
Expand Down Expand Up @@ -49,54 +51,147 @@ Check out RSS-Bridge right now on https://rss-bridge.org/bridge01/
Alternatively find another
[public instance](https://rss-bridge.github.io/rss-bridge/General/Public_Hosts.html).

Requires minimum PHP 7.4.

## Tutorial

### Install with composer or git
### How to install on traditional shared web hosting

Requires minimum PHP 7.4.
RSS-Bridge can basically be unzipped in a web folder. Should be working instantly.

```shell
apt install nginx php-fpm php-mbstring php-simplexml php-curl
```
Latest zip as of Sep 2023: https://github.com/RSS-Bridge/rss-bridge/archive/refs/tags/2023-09-24.zip

```shell
cd /var/www
composer create-project -v --no-dev rss-bridge/rss-bridge
```
### How to install on Debian 12 (nginx + php-fpm)

These instructions have been tested on a fresh Debian 12 VM from Digital Ocean (1vcpu-512mb-10gb, 5 USD/month).

```shell
timedatectl set-timezone Europe/Oslo

apt install git nginx php8.2-fpm php-mbstring php-simplexml php-curl

# Create a new user account
useradd --shell /bin/bash --create-home rss-bridge

cd /var/www
git clone https://github.com/RSS-Bridge/rss-bridge.git
```

Config:
# Create folder and change ownership
mkdir rss-bridge && chown rss-bridge:rss-bridge rss-bridge/

```shell
# Give the http user write permission to the cache folder
chown www-data:www-data /var/www/rss-bridge/cache
# Become user
su rss-bridge

# Fetch latest master
git clone https://github.com/RSS-Bridge/rss-bridge.git rss-bridge/
cd rss-bridge

# Copy over the default config
cp -v config.default.ini.php config.ini.php

# Optionally copy over the default config file
cp config.default.ini.php config.ini.php
# Give full permissions only to owner (rss-bridge)
chmod 700 -R ./

# Give read and execute to others (nginx and php-fpm)
chmod o+rx ./ ./static

# Give read to others (nginx)
chmod o+r -R ./static
```

Example config for nginx:
Nginx config:

```nginx
# /etc/nginx/sites-enabled/rssbridge
# /etc/nginx/sites-enabled/rss-bridge.conf
server {
listen 80;
server_name example.com;
root /var/www/rss-bridge;
index index.php;
access_log /var/log/nginx/rss-bridge.access.log;
error_log /var/log/nginx/rss-bridge.error.log;
# Intentionally not setting a root folder here
# autoindex is off by default but feels good to explicitly turn off
autoindex off;
location ~ \.php$ {
# Static content only served here
location /static/ {
alias /var/www/rss-bridge/static/;
}
# Pass off to php-fpm only when location is exactly /
location = / {
root /var/www/rss-bridge/;
include snippets/fastcgi-php.conf;
fastcgi_read_timeout 60s;
fastcgi_pass unix:/run/php/php-fpm.sock;
fastcgi_pass unix:/run/php/rss-bridge.sock;
}
# Reduce spam
location = /favicon.ico {
access_log off;
log_not_found off;
}
# Reduce spam
location = /robots.txt {
access_log off;
log_not_found off;
}
}
```

PHP FPM pool config:
```ini
; /etc/php/8.2/fpm/pool.d/rss-bridge.conf

[rss-bridge]

user = rss-bridge
group = rss-bridge

listen = /run/php/rss-bridge.sock

listen.owner = www-data
listen.group = www-data

pm = static
pm.max_children = 10
pm.max_requests = 500
```

PHP ini config:
```ini
; /etc/php/8.2/fpm/conf.d/30-rss-bridge.ini

max_execution_time = 20
memory_limit = 64M
```

Restart fpm and nginx:

```shell
# Lint and restart php-fpm
php-fpm8.2 -t
systemctl restart php8.2-fpm

# Lint and restart nginx
nginx -t
systemctl restart nginx
```

### How to install from Composer

Install the latest release.

```shell
cd /var/www
composer create-project -v --no-dev rss-bridge/rss-bridge
```

### How to install with Caddy

TODO. See https://github.com/RSS-Bridge/rss-bridge/issues/3785

### Install from Docker Hub:

Install by downloading the docker image from Docker Hub:
Expand Down Expand Up @@ -154,6 +249,7 @@ Browse http://localhost:3000/
[![Deploy on Scalingo](https://cdn.scalingo.com/deploy/button.svg)](https://my.scalingo.com/deploy?source=https://github.com/sebsauvage/rss-bridge)
[![Deploy to Heroku](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy)
[![Deploy to Cloudron](https://cloudron.io/img/button.svg)](https://www.cloudron.io/store/com.rssbridgeapp.cloudronapp.html)
[![Run on PikaPods](https://www.pikapods.com/static/run-button.svg)](https://www.pikapods.com/pods?run=rssbridge)

The Heroku quick deploy currently does not work. It might possibly work if you fork this repo and
modify the `repository` in `scalingo.json`. See https://github.com/RSS-Bridge/rss-bridge/issues/2688
Expand All @@ -163,6 +259,22 @@ Learn more in

## How-to

### How to fix "PHP Fatal error: Uncaught Exception: The FileCache path is not writable"

```shell
# Give rssbridge ownership
chown rssbridge:rssbridge -R /var/www/rss-bridge/cache
# Or, give www-data ownership
chown www-data:www-data -R /var/www/rss-bridge/cache
# Or, give everyone write permission
chmod 777 -R /var/www/rss-bridge/cache
# Or last ditch effort (CAREFUL)
rm -rf /var/www/rss-bridge/cache/ && mkdir /var/www/rss-bridge/cache/
```

### How to create a new bridge from scratch

Create the new bridge in e.g. `bridges/BearBlogBridge.php`:
Expand Down
9 changes: 7 additions & 2 deletions actions/DisplayAction.php
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ public function execute(array $request)
'message' => 'RSS-Bridge is down for maintenance.',
]), 503);
}

$cacheKey = 'http_' . json_encode($request);
/** @var Response $cachedResponse */
$cachedResponse = $this->cache->get($cacheKey);
Expand Down Expand Up @@ -80,16 +81,19 @@ public function execute(array $request)
$this->cache->set($cacheKey, $response, $ttl);
}

if (in_array($response->getCode(), [429, 503])) {
$this->cache->set($cacheKey, $response, 60 * 15 + rand(1, 60 * 10)); // average 20m
if (in_array($response->getCode(), [403, 429, 503])) {
// Cache these responses for about ~20 mins on average
$this->cache->set($cacheKey, $response, 60 * 15 + rand(1, 60 * 10));
}

if ($response->getCode() === 500) {
$this->cache->set($cacheKey, $response, 60 * 15);
}

if (rand(1, 100) === 2) {
$this->cache->prune();
}

return $response;
}

Expand Down Expand Up @@ -182,6 +186,7 @@ private function createFeedItemFromException($e, BridgeAbstract $bridge): FeedIt

private function logBridgeError($bridgeName, $code)
{
// todo: it's not really necessary to json encode $report
$cacheKey = 'error_reporting_' . $bridgeName . '_' . $code;
$report = $this->cache->get($cacheKey);
if ($report) {
Expand Down
2 changes: 1 addition & 1 deletion actions/FindfeedAction.php
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ public function execute(array $request)
$bridgeParams['bridge'] = $bridgeClassName;
$bridgeParams['format'] = $format;
$content = [
'url' => get_home_page_url() . '?action=display&' . http_build_query($bridgeParams),
'url' => './?action=display&' . http_build_query($bridgeParams),
'bridgeParams' => $bridgeParams,
'bridgeData' => $bridgeData,
'bridgeMeta' => [
Expand Down
1 change: 1 addition & 0 deletions actions/FrontpageAction.php
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ public function execute(array $request)
}
}

// todo: cache this renderered template
return render(__DIR__ . '/../templates/frontpage.html.php', [
'messages' => $messages,
'admin_email' => Configuration::getConfig('admin', 'email'),
Expand Down
24 changes: 17 additions & 7 deletions bridges/ARDAudiothekBridge.php
View file
Original file line number Diff line number Diff line change
Expand Up @@ -63,11 +63,13 @@ class ARDAudiothekBridge extends BridgeAbstract

public function collectData()
{
$oldTz = date_default_timezone_get();
$path = $this->getInput('path');
$limit = $this->getInput('limit');

$oldTz = date_default_timezone_get();
date_default_timezone_set('Europe/Berlin');

$pathComponents = explode('/', $this->getInput('path'));
$pathComponents = explode('/', $path);
if (empty($pathComponents)) {
returnClientError('Path may not be empty');
}
Expand All @@ -82,17 +84,21 @@ public function collectData()
}

$url = self::APIENDPOINT . 'programsets/' . $showID . '/';
$rawJSON = getContents($url);
$processedJSON = json_decode($rawJSON)->data->programSet;
$json1 = getContents($url);
$data1 = Json::decode($json1, false);
$processedJSON = $data1->data->programSet;
if (!$processedJSON) {
throw new \Exception('Unable to find show id: ' . $showID);
}

$limit = $this->getInput('limit');
$answerLength = 1;
$offset = 0;
$numberOfElements = 1;

while ($answerLength != 0 && $offset < $numberOfElements && (is_null($limit) || $offset < $limit)) {
$rawJSON = getContents($url . '?offset=' . $offset);
$processedJSON = json_decode($rawJSON)->data->programSet;
$json2 = getContents($url . '?offset=' . $offset);
$data2 = Json::decode($json2, false);
$processedJSON = $data2->data->programSet;

$answerLength = count($processedJSON->items->nodes);
$offset = $offset + $answerLength;
Expand All @@ -119,6 +125,10 @@ public function collectData()
$item['categories'] = [$category];
}

$item['itunes'] = [
'duration' => $audio->duration,
];

$this->items[] = $item;
}
}
Expand Down
28 changes: 15 additions & 13 deletions bridges/ArsTechnicaBridge.php
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,32 +35,34 @@ public function collectData()

protected function parseItem(array $item)
{
$item_html = getSimpleHTMLDOMCached($item['uri'] . '&amp');
$item_html = getSimpleHTMLDOMCached($item['uri']);
$item_html = defaultLinkTo($item_html, self::URI);
$item['content'] = $item_html->find('.amp-wp-article-content', 0);
$item['content'] = $item_html->find('.article-content', 0);

$pages = $item_html->find('nav.page-numbers > .numbers > a', -2);
if (null !== $pages) {
for ($i = 2; $i <= $pages->innertext; $i++) {
$page_url = $item['uri'] . '&page=' . $i;
$page_html = getSimpleHTMLDOMCached($page_url);
$page_html = defaultLinkTo($page_html, self::URI);
$item['content'] .= $page_html->find('.article-content', 0);
}
$item['content'] = str_get_html($item['content']);
}

// remove various ars advertising
$item['content']->find('#social-left', 0)->remove();
foreach ($item['content']->find('.ars-component-buy-box') as $ad) {
$ad->remove();
}
foreach ($item['content']->find('i-amphtml-sizer') as $ad) {
foreach ($item['content']->find('.ad_wrapper') as $ad) {
$ad->remove();
}
foreach ($item['content']->find('.sidebar') as $ad) {
$ad->remove();
}

foreach ($item['content']->find('a') as $link) { //remove amp redirect links
$url = $link->getAttribute('href');
if (str_contains($url, 'go.redirectingat.com')) {
$url = extractFromDelimiters($url, 'url=', '&amp');
$url = urldecode($url);
$link->setAttribute('href', $url);
}
}

$item['content'] = backgroundToImg(str_replace('data-amp-original-style="background-image', 'style="background-image', $item['content']));
$item['content'] = backgroundToImg($item['content']);

$item['uid'] = explode('=', $item['uri'])[1];

Expand Down
Loading

0 comments on commit 3733bc1

Please sign in to comment.