Bump version

* Implement initial background tasks concept

* fix property reference

* update requirements.txt

* simplify bookmark null check

* improve web archive url display

* add background tasks test

* add basic supervisor setup

* schedule missing snapshot creation on login

* remove task locks and clear task history before starting background task processor

* batch create snapshots after import

* fix script reference in supervisord.conf

* add option to disable background tasks

* restructure feature overview

.dockerignore

@@ -7,6 +7,7 @@
 /docs
 /static
 /build
+/out
 
 /.dockerignore
 /.gitignore
@@ -17,10 +18,13 @@
 /*.patch
 /*.md
 /*.js
+/*.log
+/*.pid
 
 # Whitelist files needed in build or prod image
 !/rollup.config.js
 !/bootstrap.sh
+!/background-tasks-wrapper.sh
 
 # Remove development settings
 /siteroot/settings/dev.py

.env.sample

@@ -5,5 +5,7 @@ LD_HOST_PORT=9090
 # Directory on the host system that should be mounted as data dir into the Docker container
 LD_HOST_DATA_DIR=./data
 
+# Option to disable background tasks
+LD_DISABLE_BACKGROUND_TASKS=False
 # Option to disable URL validation for bookmarks completely
-LD_DISABLE_URL_VALIDATION=False
+LD_DISABLE_URL_VALIDATION=False

CHANGELOG.md

@@ -1,5 +1,44 @@
 # Changelog
 
+## v1.8.2 (02/10/2021)
+- [**bug**] Fix jumping search box [#163](https://github.com/sissbruecker/linkding/pull/163)
+---
+
+## v1.8.1 (01/10/2021)
+- [**enhancement**] Add global shortcut for search [#161](https://github.com/sissbruecker/linkding/pull/161)
+  - allows to press `s` to focus the search input
+
+---
+
+## v1.8.0 (04/09/2021)
+- [**enhancement**] Wayback Machine Integration [#59](https://github.com/sissbruecker/linkding/issues/59)
+  - Automatically creates snapshots of bookmarked websites on [web archive](https://archive.org/web/)
+  - This is one of the largest changes yet and adds a task processor that runs as a separate process in the background. If you run into issues with this feature, it can be disabled using the [LD_DISABLE_BACKGROUND_TASKS](https://github.com/sissbruecker/linkding/blob/master/docs/Options.md#ld_disable_background_tasks) option
+
+---
+
+## v1.7.2 (26/08/2021)
+- [**enhancement**] Add support for nanosecond resolution timestamps for bookmark import (e.g. Google Bookmarks) [#146](https://github.com/sissbruecker/linkding/issues/146)
+
+---
+
+## v1.7.1 (25/08/2021)
+- [**bug**] umlaut/non-ascii characters broken when using bookmarklet (firefox) [#148](https://github.com/sissbruecker/linkding/issues/148)
+- [**bug**] Bookmark import accepts empty URL values [#124](https://github.com/sissbruecker/linkding/issues/124)
+- [**enhancement**] Show the version in the settings [#104](https://github.com/sissbruecker/linkding/issues/104)
+
+---
+
+## v1.7.0 (17/08/2021)
+- Upgrade to Django 3
+- Bump other dependencies
+
+---
+
+## v1.6.5 (15/08/2021)
+- [**enhancement**] query with multiple hashtags very slow [#112](https://github.com/sissbruecker/linkding/issues/112)
+---
+
 ## v1.6.4 (13/05/2021)
 - Update dependencies for security fixes
 

Dockerfile

@@ -9,7 +9,7 @@ COPY . .
 RUN npm run build
 
 
-FROM python:3.9-slim AS python-base
+FROM python:3.9.6-slim-buster AS python-base
 RUN apt-get update && apt-get -y install build-essential
 WORKDIR /etc/linkding
 
@@ -33,7 +33,7 @@ RUN mkdir /opt/venv && \
     /opt/venv/bin/pip install -Ur requirements.txt
 
 
-FROM python:3.9-slim as final
+FROM python:3.9.6-slim-buster as final
 RUN apt-get update && apt-get -y install mime-support
 WORKDIR /etc/linkding
 # copy prod dependencies
@@ -47,6 +47,8 @@ EXPOSE 9090
 # Activate virtual env
 ENV VIRTUAL_ENV /opt/venv
 ENV PATH /opt/venv/bin:$PATH
+# Allow running containers as an an arbitrary user in the root group, to support deployment scenarios like OpenShift, Podman
+RUN ["chmod", "g+w", "."]
 # Run bootstrap logic
 RUN ["chmod", "+x", "./bootstrap.sh"]
 CMD ["./bootstrap.sh"]

README.md

@@ -13,17 +13,14 @@ The name comes from:
 - Search by text or tags
 - Bulk editing
 - Bookmark archive
-- Automatically provides titles and descriptions from linked websites 
-- Import and export bookmarks in Netscape HTML format
-- Extensions for [Firefox](https://addons.mozilla.org/de/firefox/addon/linkding-extension/) and [Chrome](https://chrome.google.com/webstore/detail/linkding-extension/beakmhbijpdhipnjhnclmhgjlddhidpe)
-- Bookmarklet that should work in most browsers
 - Dark mode
-- Easy to set up using Docker
-- Uses SQLite as database
-- Works without Javascript
-- ...but has several UI enhancements when Javascript is enabled
+- Automatically creates snapshots of bookmarked websites on [web archive](https://archive.org/web/)
+- Automatically provides titles and descriptions of bookmarked websites 
+- Import and export bookmarks in Netscape HTML format
+- Extensions for [Firefox](https://addons.mozilla.org/de/firefox/addon/linkding-extension/) and [Chrome](https://chrome.google.com/webstore/detail/linkding-extension/beakmhbijpdhipnjhnclmhgjlddhidpe), and a bookmarklet that should work in most browsers
 - REST API for developing 3rd party apps
 - Admin panel for user self-service and raw data access
+- Easy to set up using Docker, uses SQLite as database
 
 
 **Demo:** https://demo.linkding.link/ (configured with open registration)
@@ -88,7 +85,7 @@ If you can not or don't want to use Docker you can install the application manua
 
 ### Hosting
 
-The application runs in a web-server called [uWSGI](https://uwsgi-docs.readthedocs.io/en/latest/) that is production-ready and that you can expose to the web. If you don't know how to configure your server to expose the application to the web there are several more steps involved. I can not support support the process here, but I can give some pointers on what to search for:
+The application runs in a web-server called [uWSGI](https://uwsgi-docs.readthedocs.io/en/latest/) that is production-ready and that you can expose to the web. If you don't know how to configure your server to expose the application to the web there are several more steps involved. I can not support the process here, but I can give some pointers on what to search for:
 - first get the app running (described in this document)
 - open the port that the application is running on in your servers firewall
 - depending on your network configuration, forward the opened port in your network router, so that the application can be addressed from the internet using your public IP address and the opened port
@@ -126,7 +123,7 @@ Note that any proxy servers that you are running in front of linkding may have t
 
 ## Development
 
-The application is open source, so you are free to modify or contribute. The application is built using the Django web framework. You can get started by checking out the excellent Django docs: https://docs.djangoproject.com/en/3.0/. The `bookmarks` folder contains the actual bookmark application, `siteroot` is the Django root application. Other than that the code should be self-explanatory / standard Django stuff 🙂.
+The application is open source, so you are free to modify or contribute. The application is built using the Django web framework. You can get started by checking out the excellent Django docs: https://docs.djangoproject.com/en/3.2/. The `bookmarks` folder contains the actual bookmark application, `siteroot` is the Django root application. Other than that the code should be self-explanatory / standard Django stuff 🙂.
 
 ### Prerequisites
 - Python 3

background-tasks-wrapper.sh

@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+# Wrapper script used by supervisord to first clear task locks before starting the background task processor
+
+python manage.py clean_tasks
+exec python manage.py process_tasks

bookmarks/admin.py

@@ -5,7 +5,7 @@ from django.contrib.auth.models import User
 from django.db.models import Count, QuerySet
 from django.utils.translation import ngettext, gettext
 from rest_framework.authtoken.admin import TokenAdmin
-from rest_framework.authtoken.models import Token
+from rest_framework.authtoken.models import TokenProxy
 
 from bookmarks.models import Bookmark, Tag, UserProfile
 from bookmarks.services.bookmarks import archive_bookmark, unarchive_bookmark
@@ -97,4 +97,4 @@ linkding_admin_site = LinkdingAdminSite()
 linkding_admin_site.register(Bookmark, AdminBookmark)
 linkding_admin_site.register(Tag, AdminTag)
 linkding_admin_site.register(User, AdminCustomUser)
-linkding_admin_site.register(Token, TokenAdmin)
+linkding_admin_site.register(TokenProxy, TokenAdmin)

bookmarks/apps.py

@@ -3,3 +3,7 @@ from django.apps import AppConfig
 
 class BookmarksConfig(AppConfig):
     name = 'bookmarks'
+
+    def ready(self):
+        # Register signal handlers
+        import bookmarks.signals

bookmarks/management/commands/clean_tasks.py

@@ -0,0 +1,15 @@
+from background_task.models import Task, CompletedTask
+from django.core.management.base import BaseCommand
+
+
+class Command(BaseCommand):
+    help = "Remove task locks and clear completed task history"
+
+    def handle(self, *args, **options):
+        # Remove task locks
+        # If the background task processor exited while executing tasks, these tasks would still be marked as locked,
+        # even though no process is working on them, and would prevent the task processor from picking the next task in
+        # the queue
+        Task.objects.all().update(locked_by=None, locked_at=None)
+        # Clear task history to prevent them from bloating the DB
+        CompletedTask.objects.all().delete()

bookmarks/migrations/0009_bookmark_web_archive_snapshot_url.py

@@ -0,0 +1,18 @@
+# Generated by Django 2.2.20 on 2021-05-16 14:35
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('bookmarks', '0008_userprofile_bookmark_date_display'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='bookmark',
+            name='web_archive_snapshot_url',
+            field=models.CharField(blank=True, max_length=2048),
+        ),
+    ]

bookmarks/migrations/0010_userprofile_bookmark_link_target.py

@@ -0,0 +1,18 @@
+# Generated by Django 3.2.6 on 2021-10-03 06:35
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('bookmarks', '0009_bookmark_web_archive_snapshot_url'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='userprofile',
+            name='bookmark_link_target',
+            field=models.CharField(choices=[('_blank', 'New page'), ('_self', 'Same page')], default='_blank', max_length=10),
+        ),
+    ]

bookmarks/models.py

@@ -41,6 +41,7 @@ class Bookmark(models.Model):
     description = models.TextField(blank=True)
     website_title = models.CharField(max_length=512, blank=True, null=True)
     website_description = models.TextField(blank=True, null=True)
+    web_archive_snapshot_url = models.CharField(max_length=2048, blank=True)
     unread = models.BooleanField(default=True)
     is_archived = models.BooleanField(default=False)
     date_added = models.DateTimeField()
@@ -115,16 +116,24 @@ class UserProfile(models.Model):
         (BOOKMARK_DATE_DISPLAY_ABSOLUTE, 'Absolute'),
         (BOOKMARK_DATE_DISPLAY_HIDDEN, 'Hidden'),
     ]
+    BOOKMARK_LINK_TARGET_BLANK = '_blank'
+    BOOKMARK_LINK_TARGET_SELF = '_self'
+    BOOKMARK_LINK_TARGET_CHOICES = [
+        (BOOKMARK_LINK_TARGET_BLANK, 'New page'),
+        (BOOKMARK_LINK_TARGET_SELF, 'Same page'),
+    ]
     user = models.OneToOneField(get_user_model(), related_name='profile', on_delete=models.CASCADE)
     theme = models.CharField(max_length=10, choices=THEME_CHOICES, blank=False, default=THEME_AUTO)
     bookmark_date_display = models.CharField(max_length=10, choices=BOOKMARK_DATE_DISPLAY_CHOICES, blank=False,
                                              default=BOOKMARK_DATE_DISPLAY_RELATIVE)
+    bookmark_link_target = models.CharField(max_length=10, choices=BOOKMARK_LINK_TARGET_CHOICES, blank=False,
+                                            default=BOOKMARK_LINK_TARGET_BLANK)
 
 
 class UserProfileForm(forms.ModelForm):
     class Meta:
         model = UserProfile
-        fields = ['theme', 'bookmark_date_display']
+        fields = ['theme', 'bookmark_date_display', 'bookmark_link_target']
 
 
 @receiver(post_save, sender=get_user_model())

bookmarks/services/bookmarks.py

@@ -6,6 +6,7 @@ from django.utils import timezone
 from bookmarks.models import Bookmark, parse_tag_string
 from bookmarks.services.tags import get_or_create_tags
 from bookmarks.services.website_loader import load_website_metadata
+from bookmarks.services import tasks
 
 
 def create_bookmark(bookmark: Bookmark, tag_string: str, current_user: User):
@@ -27,10 +28,16 @@ def create_bookmark(bookmark: Bookmark, tag_string: str, current_user: User):
     # Update tag list
     _update_bookmark_tags(bookmark, tag_string, current_user)
     bookmark.save()
+    # Create snapshot on web archive
+    tasks.create_web_archive_snapshot(bookmark.id, False)
+
     return bookmark
 
 
 def update_bookmark(bookmark: Bookmark, tag_string, current_user: User):
+    # Detect URL change
+    original_bookmark = Bookmark.objects.get(id=bookmark.id)
+    has_url_changed = original_bookmark.url != bookmark.url
     # Update website info
     _update_website_metadata(bookmark)
     # Update tag list
@@ -38,6 +45,10 @@ def update_bookmark(bookmark: Bookmark, tag_string, current_user: User):
     # Update dates
     bookmark.date_modified = timezone.now()
     bookmark.save()
+    # Update web archive snapshot, if URL changed
+    if has_url_changed:
+        tasks.create_web_archive_snapshot(bookmark.id, True)
+
     return bookmark
 
 

bookmarks/services/importer.py

@@ -1,13 +1,14 @@
 import logging
 from dataclasses import dataclass
-from datetime import datetime
 
 from django.contrib.auth.models import User
 from django.utils import timezone
 
 from bookmarks.models import Bookmark, parse_tag_string
+from bookmarks.services import tasks
 from bookmarks.services.parser import parse, NetscapeBookmark
 from bookmarks.services.tags import get_or_create_tags
+from bookmarks.utils import parse_timestamp
 
 logger = logging.getLogger(__name__)
 
@@ -38,6 +39,9 @@ def import_netscape_html(html: str, user: User):
             logging.exception('Error importing bookmark: ' + shortened_bookmark_tag_str)
             result.failed = result.failed + 1
 
+    # Create snapshots for newly imported bookmarks
+    tasks.schedule_bookmarks_without_snapshots(user.id)
+
     return result
 
 
@@ -47,7 +51,7 @@ def _import_bookmark_tag(netscape_bookmark: NetscapeBookmark, user: User):
 
     bookmark.url = netscape_bookmark.href
     if netscape_bookmark.date_added:
-        bookmark.date_added = datetime.utcfromtimestamp(int(netscape_bookmark.date_added)).astimezone()
+        bookmark.date_added = parse_timestamp(netscape_bookmark.date_added)
     else:
         bookmark.date_added = timezone.now()
     bookmark.date_modified = bookmark.date_added
@@ -57,6 +61,7 @@ def _import_bookmark_tag(netscape_bookmark: NetscapeBookmark, user: User):
         bookmark.description = netscape_bookmark.description
     bookmark.owner = user
 
+    bookmark.full_clean()
     bookmark.save()
 
     # Set tags

bookmarks/services/tasks.py

@@ -0,0 +1,62 @@
+import logging
+
+import waybackpy
+from background_task import background
+from django.conf import settings
+from django.contrib.auth import get_user_model
+from waybackpy.exceptions import WaybackError
+
+from bookmarks.models import Bookmark
+
+logger = logging.getLogger(__name__)
+
+
+def when_background_tasks_enabled(fn):
+    def wrapper(*args, **kwargs):
+        if settings.LD_DISABLE_BACKGROUND_TASKS:
+            return
+        return fn(*args, **kwargs)
+
+    # Expose attributes from wrapped TaskProxy function
+    attrs = vars(fn)
+    for key, value in attrs.items():
+        setattr(wrapper, key, value)
+
+    return wrapper
+
+
+@when_background_tasks_enabled
+@background()
+def create_web_archive_snapshot(bookmark_id: int, force_update: bool):
+    try:
+        bookmark = Bookmark.objects.get(id=bookmark_id)
+    except Bookmark.DoesNotExist:
+        return
+
+    # Skip if snapshot exists and update is not explicitly requested
+    if bookmark.web_archive_snapshot_url and not force_update:
+        return
+
+    logger.debug(f'Create web archive link for bookmark: {bookmark}...')
+
+    wayback = waybackpy.Url(bookmark.url)
+
+    try:
+        archive = wayback.save()
+    except WaybackError as error:
+        logger.exception(f'Error creating web archive link for bookmark: {bookmark}...', exc_info=error)
+        raise
+
+    bookmark.web_archive_snapshot_url = archive.archive_url
+    bookmark.save()
+    logger.debug(f'Successfully created web archive link for bookmark: {bookmark}...')
+
+
+@when_background_tasks_enabled
+@background()
+def schedule_bookmarks_without_snapshots(user_id: int):
+    user = get_user_model().objects.get(id=user_id)
+    bookmarks_without_snapshots = Bookmark.objects.filter(web_archive_snapshot_url__exact='', owner=user)
+
+    for bookmark in bookmarks_without_snapshots:
+        create_web_archive_snapshot(bookmark.id, False)

bookmarks/services/website_loader.py

@@ -2,6 +2,7 @@ from dataclasses import dataclass
 
 import requests
 from bs4 import BeautifulSoup
+from charset_normalizer import from_bytes
 
 
 @dataclass
@@ -33,5 +34,11 @@ def load_website_metadata(url: str):
 
 
 def load_page(url: str):
-    r = requests.get(url)
-    return r.text
+    r = requests.get(url, timeout=10)
+
+    # Use charset_normalizer to determine encoding that best matches the response content
+    # Several sites seem to specify the response encoding incorrectly, so we ignore it and use custom logic instead
+    # This is different from Response.text which does respect the encoding specified in the response first,
+    # before trying to determine one
+    results = from_bytes(r.content)
+    return str(results.best())

bookmarks/signals.py

@@ -0,0 +1,8 @@
+from django.contrib.auth import user_logged_in
+from django.dispatch import receiver
+from bookmarks.services import tasks
+
+
+@receiver(user_logged_in)
+def user_logged_in(sender, request, user, **kwargs):
+    tasks.schedule_bookmarks_without_snapshots(user.id)

bookmarks/static/shared.js

@@ -40,6 +40,28 @@
         });
     }
 
+    function initGlobalShortcuts() {
+        // Focus search button
+        document.addEventListener('keydown', function (event) {
+            // Filter for shortcut key
+            if (event.key !== 's') return;
+            // Skip if event occurred within an input element
+            const targetNodeName = event.target.nodeName;
+            const isInputTarget = targetNodeName === 'INPUT'
+                || targetNodeName === 'SELECT'
+                || targetNodeName === 'TEXTAREA';
 
-    initConfirmationButtons()
+            if (isInputTarget) return;
+
+            const searchInput = document.querySelector('input[type="search"]');
+
+            if (searchInput) {
+                searchInput.focus();
+                event.preventDefault();
+            }
+        });
+    }
+
+    initConfirmationButtons();
+    initGlobalShortcuts();
 })()

bookmarks/styles/bookmarks.scss

@@ -1,14 +1,16 @@
 .bookmarks-page .search {
+  $searchbox-width: 180px;
+  $searchbox-width-md: 300px;
   $searchbox-height: 1.8rem;
 
   // Regular input
   input[type='search'] {
-    width: 180px;
+    width: $searchbox-width;
     height: $searchbox-height;
     -webkit-appearance: none;
 
     @media (min-width: $control-width-md) {
-      width: 300px;
+      width: $searchbox-width-md;
     }
   }
 
@@ -18,14 +20,19 @@
     height: $searchbox-height;
 
     .form-autocomplete-input {
+      width: $searchbox-width;
       height: $searchbox-height;
-      width: 100%;
 
       input[type='search'] {
+        width: 100%;
         height: 100%;
         margin: 0;
         border: none;
       }
+
+      @media (min-width: $control-width-md) {
+        width: $searchbox-width-md;
+      }
     }
   }
 }
@@ -54,6 +61,10 @@ ul.bookmark-list {
     margin-right: 0.1rem;
   }
 
+  .actions .date-label a {
+    color: $gray-color;
+  }
+
   .actions .btn-link {
     color: $gray-color;
     padding: 0;

bookmarks/templates/bookmarks/archive.html

@@ -26,7 +26,7 @@
                 {% if empty %}
                     {% include 'bookmarks/empty_bookmarks.html' %}
                 {% else %}
-                    {% bookmark_list bookmarks return_url %}
+                    {% bookmark_list bookmarks return_url link_target %}
                 {% endif %}
             </form>
         </section>

bookmarks/templates/bookmarks/bookmark_list.html

@@ -9,7 +9,7 @@
                 <i class="form-icon"></i>
             </label>
             <div class="title truncate">
-                <a href="{{ bookmark.url }}" target="_blank" rel="noopener">{{ bookmark.resolved_title }}</a>
+                <a href="{{ bookmark.url }}" target="{{ link_target }}" rel="noopener">{{ bookmark.resolved_title }}</a>
             </div>
             <div class="description truncate">
                 {% if bookmark.tag_names %}
@@ -27,11 +27,31 @@
             </div>
             <div class="actions">
                 {% if request.user.profile.bookmark_date_display == 'relative' %}
-                    <span class="text-gray text-sm">{{ bookmark.date_added|humanize_relative_date }}</span>
+                    <span class="date-label text-gray text-sm">
+                        {% if bookmark.web_archive_snapshot_url %}
+                        <a href="{{ bookmark.web_archive_snapshot_url }}"
+                           title="Show snapshot on web archive" target="{{ link_target }}" rel="noopener">
+                        {% endif %}
+                        <span>{{ bookmark.date_added|humanize_relative_date }}</span>
+                        {% if bookmark.web_archive_snapshot_url %}
+                        <span>∞</span>
+                        </a>
+                        {% endif %}
+                    </span>
                     <span class="text-gray text-sm">|</span>
                 {% endif %}
                 {% if request.user.profile.bookmark_date_display == 'absolute' %}
-                    <span class="text-gray text-sm">{{ bookmark.date_added|humanize_absolute_date }}</span>
+                    <span class="date-label text-gray text-sm">
+                        {% if bookmark.web_archive_snapshot_url %}
+                        <a href="{{ bookmark.web_archive_snapshot_url }}"
+                           title="Show snapshot on web archive" target="{{ link_target }}" rel="noopener">
+                        {% endif %}
+                        <span>{{ bookmark.date_added|humanize_absolute_date }}</span>
+                        {% if bookmark.web_archive_snapshot_url %}
+                        <span>∞</span>
+                        </a>
+                        {% endif %}
+                    </span>
                     <span class="text-gray text-sm">|</span>
                 {% endif %}
                 <a href="{% url 'bookmarks:edit' bookmark.id %}?return_url={{ return_url }}"

bookmarks/templates/bookmarks/index.html

@@ -26,7 +26,7 @@
                 {% if empty %}
                     {% include 'bookmarks/empty_bookmarks.html' %}
                 {% else %}
-                    {% bookmark_list bookmarks return_url %}
+                    {% bookmark_list bookmarks return_url link_target %}
                 {% endif %}
             </form>
         </section>

bookmarks/templates/settings/api.html

@@ -17,7 +17,7 @@
             </div>
             <p><strong>Please treat this token as you would any other credential.</strong> Any party with access to this
                 token can access and manage all your bookmarks.</p>
-            <p>If you think that a token was compromised you can revoke (delete) it in the <a href="{% url 'admin:authtoken_token_changelist' %}">admin panel</a>. After deleting the token, a new one will be generated when you reload this settings page.</p>
+            <p>If you think that a token was compromised you can revoke (delete) it in the <a href="{% url 'admin:authtoken_tokenproxy_changelist' %}">admin panel</a>. After deleting the token, a new one will be generated when you reload this settings page.</p>
         </section>
 
     </div>

bookmarks/templates/settings/general.html

@@ -19,6 +19,10 @@
                     <label for="{{ form.bookmark_date_display.id_for_label }}" class="form-label">Bookmark date format</label>
                     {{ form.bookmark_date_display|add_class:"form-select col-2 col-sm-12" }}
                 </div>
+                <div class="form-group">
+                    <label for="{{ form.bookmark_link_target.id_for_label }}" class="form-label">Open bookmarks in</label>
+                    {{ form.bookmark_link_target|add_class:"form-select col-2 col-sm-12" }}
+                </div>
                 <div class="form-group">
                     <input type="submit" value="Save" class="btn btn-primary mt-2">
                 </div>
@@ -69,6 +73,15 @@
             {% endif %}
         </section>
 
+        {# About section #}
+        <section class="content-area">
+            <h2>About</h2>
+            <p>Version: {{ app_version }}</p>
+            <p>
+                Code: <a href="https://github.com/sissbruecker/linkding/"
+                    target="_blank">GitHub</a>
+            </p>
+        </section>
     </div>
 
 {% endblock %}

bookmarks/templatetags/bookmarks.py

@@ -51,11 +51,12 @@ def tag_cloud(context, tags: List[Tag]):
 
 
 @register.inclusion_tag('bookmarks/bookmark_list.html', name='bookmark_list', takes_context=True)
-def bookmark_list(context, bookmarks: Page, return_url: str):
+def bookmark_list(context, bookmarks: Page, return_url: str, link_target: str = '_blank'):
     return {
         'request': context['request'],
         'bookmarks': bookmarks,
-        'return_url': return_url
+        'return_url': return_url,
+        'link_target': link_target,
     }
 
 

bookmarks/tests/helpers.py

@@ -1,4 +1,5 @@
 import random
+import logging
 
 from django.contrib.auth.models import User
 from django.utils import timezone
@@ -27,6 +28,7 @@ class BookmarkFactoryMixin:
                        description: str = '',
                        website_title: str = '',
                        website_description: str = '',
+                       web_archive_snapshot_url: str = '',
                        ):
         if tags is None:
             tags = []
@@ -44,7 +46,8 @@ class BookmarkFactoryMixin:
             date_added=timezone.now(),
             date_modified=timezone.now(),
             owner=user,
-            is_archived=is_archived
+            is_archived=is_archived,
+            web_archive_snapshot_url=web_archive_snapshot_url,
         )
         bookmark.save()
         for tag in tags:
@@ -117,3 +120,14 @@ def random_sentence(num_words: int = None, including_word: str = ''):
     random.shuffle(selected_words)
 
     return ' '.join(selected_words)
+
+
+def disable_logging(f):
+    def wrapper(*args):
+        logging.disable(logging.CRITICAL)
+        result = f(*args)
+        logging.disable(logging.NOTSET)
+
+        return result
+
+    return wrapper

bookmarks/tests/resources/simple_valid_import_file.html

@@ -11,10 +11,10 @@
     <DT><A HREF="https://example.com/1" ADD_DATE="1616337559" PRIVATE="0" TOREAD="0" TAGS="tag1">test title 1</A>
     <DD>test description 1
 
-    <DT><A HREF="https://example.com/2" ADD_DATE="1616337559" PRIVATE="0" TOREAD="0" TAGS="tag2">test title 2</A>
+    <DT><A HREF="https://example.com/2" ADD_DATE="1616337559000" PRIVATE="0" TOREAD="0" TAGS="tag2">test title 2</A>
     <DD>test description 2
 
-    <DT><A HREF="https://example.com/3" ADD_DATE="1616337559" PRIVATE="0" TOREAD="0" TAGS="tag3">test title 3</A>
+    <DT><A HREF="https://example.com/3" ADD_DATE="1616337559000000" PRIVATE="0" TOREAD="0" TAGS="tag3">test title 3</A>
     <DD>test description 3
 
 </DL><p>

bookmarks/tests/test_bookmark_archived_view.py

@@ -2,7 +2,7 @@ from django.contrib.auth.models import User
 from django.test import TestCase
 from django.urls import reverse
 
-from bookmarks.models import Bookmark, Tag
+from bookmarks.models import Bookmark, Tag, UserProfile
 from bookmarks.tests.helpers import BookmarkFactoryMixin
 
 
@@ -12,22 +12,22 @@ class BookmarkArchivedViewTestCase(TestCase, BookmarkFactoryMixin):
         user = self.get_or_create_test_user()
         self.client.force_login(user)
 
-    def assertVisibleBookmarks(self, response, bookmarks: [Bookmark]):
+    def assertVisibleBookmarks(self, response, bookmarks: [Bookmark], link_target: str = '_blank'):
         html = response.content.decode()
         self.assertContains(response, 'data-is-bookmark-item', count=len(bookmarks))
 
         for bookmark in bookmarks:
             self.assertInHTML(
-                '<a href="{0}" target="_blank" rel="noopener">{1}</a>'.format(bookmark.url, bookmark.resolved_title),
+                f'<a href="{bookmark.url}" target="{link_target}" rel="noopener">{bookmark.resolved_title}</a>',
                 html
             )
 
-    def assertInvisibleBookmarks(self, response, bookmarks: [Bookmark]):
+    def assertInvisibleBookmarks(self, response, bookmarks: [Bookmark], link_target: str = '_blank'):
         html = response.content.decode()
 
         for bookmark in bookmarks:
             self.assertInHTML(
-                '<a href="{0}" target="_blank" rel="noopener">{1}</a>'.format(bookmark.url, bookmark.resolved_title),
+                f'<a href="{bookmark.url}" target="{link_target}" rel="noopener">{bookmark.resolved_title}</a>',
                 html,
                 count=0
             )
@@ -130,3 +130,29 @@ class BookmarkArchivedViewTestCase(TestCase, BookmarkFactoryMixin):
 
         self.assertVisibleTags(response, visible_tags)
         self.assertInvisibleTags(response, invisible_tags)
+
+    def test_should_open_bookmarks_in_new_page_by_default(self):
+        visible_bookmarks = [
+            self.setup_bookmark(is_archived=True),
+            self.setup_bookmark(is_archived=True),
+            self.setup_bookmark(is_archived=True)
+        ]
+
+        response = self.client.get(reverse('bookmarks:archived'))
+
+        self.assertVisibleBookmarks(response, visible_bookmarks, '_blank')
+
+    def test_should_open_bookmarks_in_same_page_if_specified_in_user_profile(self):
+        user = self.get_or_create_test_user()
+        user.profile.bookmark_link_target = UserProfile.BOOKMARK_LINK_TARGET_SELF
+        user.profile.save()
+
+        visible_bookmarks = [
+            self.setup_bookmark(is_archived=True),
+            self.setup_bookmark(is_archived=True),
+            self.setup_bookmark(is_archived=True)
+        ]
+
+        response = self.client.get(reverse('bookmarks:archived'))
+
+        self.assertVisibleBookmarks(response, visible_bookmarks, '_self')

bookmarks/tests/test_bookmark_index_view.py

@@ -2,7 +2,7 @@ from django.contrib.auth.models import User
 from django.test import TestCase
 from django.urls import reverse
 
-from bookmarks.models import Bookmark, Tag
+from bookmarks.models import Bookmark, Tag, UserProfile
 from bookmarks.tests.helpers import BookmarkFactoryMixin
 
 
@@ -12,22 +12,22 @@ class BookmarkIndexViewTestCase(TestCase, BookmarkFactoryMixin):
         user = self.get_or_create_test_user()
         self.client.force_login(user)
 
-    def assertVisibleBookmarks(self, response, bookmarks: [Bookmark]):
+    def assertVisibleBookmarks(self, response, bookmarks: [Bookmark], link_target: str = '_blank'):
         html = response.content.decode()
         self.assertContains(response, 'data-is-bookmark-item', count=len(bookmarks))
 
         for bookmark in bookmarks:
             self.assertInHTML(
-                '<a href="{0}" target="_blank" rel="noopener">{1}</a>'.format(bookmark.url, bookmark.resolved_title),
+                f'<a href="{bookmark.url}" target="{link_target}" rel="noopener">{bookmark.resolved_title}</a>',
                 html
             )
 
-    def assertInvisibleBookmarks(self, response, bookmarks: [Bookmark]):
+    def assertInvisibleBookmarks(self, response, bookmarks: [Bookmark], link_target: str = '_blank'):
         html = response.content.decode()
 
         for bookmark in bookmarks:
             self.assertInHTML(
-                '<a href="{0}" target="_blank" rel="noopener">{1}</a>'.format(bookmark.url, bookmark.resolved_title),
+                f'<a href="{bookmark.url}" target="{link_target}" rel="noopener">{bookmark.resolved_title}</a>',
                 html,
                 count=0
             )
@@ -130,3 +130,29 @@ class BookmarkIndexViewTestCase(TestCase, BookmarkFactoryMixin):
 
         self.assertVisibleTags(response, visible_tags)
         self.assertInvisibleTags(response, invisible_tags)
+
+    def test_should_open_bookmarks_in_new_page_by_default(self):
+        visible_bookmarks = [
+            self.setup_bookmark(),
+            self.setup_bookmark(),
+            self.setup_bookmark()
+        ]
+
+        response = self.client.get(reverse('bookmarks:index'))
+
+        self.assertVisibleBookmarks(response, visible_bookmarks, '_blank')
+
+    def test_should_open_bookmarks_in_same_page_if_specified_in_user_profile(self):
+        user = self.get_or_create_test_user()
+        user.profile.bookmark_link_target = UserProfile.BOOKMARK_LINK_TARGET_SELF
+        user.profile.save()
+
+        visible_bookmarks = [
+            self.setup_bookmark(),
+            self.setup_bookmark(),
+            self.setup_bookmark()
+        ]
+
+        response = self.client.get(reverse('bookmarks:index'))
+
+        self.assertVisibleBookmarks(response, visible_bookmarks, '_self')

bookmarks/tests/test_bookmark_validation.py

@@ -34,6 +34,27 @@ class BookmarkValidationTestCase(TestCase):
     def setUp(self) -> None:
         self.user = User.objects.create_user('testuser', 'test@example.com', 'password123')
 
+    def test_bookmark_model_should_not_allow_missing_url(self):
+        bookmark = Bookmark(
+            date_added=datetime.datetime.now(),
+            date_modified=datetime.datetime.now(),
+            owner=self.user
+        )
+
+        with self.assertRaises(ValidationError):
+            bookmark.full_clean()
+
+    def test_bookmark_model_should_not_allow_empty_url(self):
+        bookmark = Bookmark(
+            url='',
+            date_added=datetime.datetime.now(),
+            date_modified=datetime.datetime.now(),
+            owner=self.user
+        )
+
+        with self.assertRaises(ValidationError):
+            bookmark.full_clean()
+
     @override_settings(LD_DISABLE_URL_VALIDATION=False)
     def test_bookmark_model_should_validate_url_if_not_disabled_in_settings(self):
         self._run_bookmark_model_url_validity_checks(ENABLED_URL_VALIDATION_TEST_CASES)

bookmarks/tests/test_bookmarks_list_tag.py

@@ -4,13 +4,39 @@ from django.template import Template, RequestContext
 from django.test import TestCase, RequestFactory
 from django.utils import timezone, formats
 
-from bookmarks.models import UserProfile
+from bookmarks.models import Bookmark, UserProfile
 from bookmarks.tests.helpers import BookmarkFactoryMixin
 
 
 class BookmarkListTagTest(TestCase, BookmarkFactoryMixin):
 
-    def render_template(self, bookmarks) -> str:
+    def assertBookmarksLink(self, html: str, bookmark: Bookmark, link_target: str = '_blank'):
+        self.assertInHTML(
+            f'<a href="{bookmark.url}" target="{link_target}" rel="noopener">{bookmark.resolved_title}</a>',
+            html
+        )
+
+    def assertDateLabel(self, html: str, label_content: str):
+        self.assertInHTML(f'''
+        <span class="date-label text-gray text-sm">
+            <span>{label_content}</span>
+        </span>
+        <span class="text-gray text-sm">|</span>
+        ''', html)
+
+    def assertWebArchiveLink(self, html: str, label_content: str, url: str, link_target: str = '_blank'):
+        self.assertInHTML(f'''
+        <span class="date-label text-gray text-sm">
+            <a href="{url}"
+               title="Show snapshot on web archive" target="{link_target}" rel="noopener">
+                <span>{label_content}</span>
+                <span>∞</span>
+            </a>
+        </span>
+        <span class="text-gray text-sm">|</span>
+        ''', html)
+
+    def render_template(self, bookmarks: [Bookmark], template: Template) -> str:
         rf = RequestFactory()
         request = rf.get('/test')
         request.user = self.get_or_create_test_user()
@@ -18,15 +44,28 @@ class BookmarkListTagTest(TestCase, BookmarkFactoryMixin):
         page = paginator.page(1)
 
         context = RequestContext(request, {'bookmarks': page, 'return_url': '/test'})
-        template_to_render = Template(
+        return template.render(context)
+
+    def render_default_template(self, bookmarks: [Bookmark]) -> str:
+        template = Template(
             '{% load bookmarks %}'
             '{% bookmark_list bookmarks return_url %}'
         )
-        return template_to_render.render(context)
+        return self.render_template(bookmarks, template)
 
-    def setup_date_format_test(self, date_display_setting):
+    def render_template_with_link_target(self, bookmarks: [Bookmark], link_target: str) -> str:
+        template = Template(
+            f'''
+            {{% load bookmarks %}}
+            {{% bookmark_list bookmarks return_url '{link_target}' %}}
+            '''
+        )
+        return self.render_template(bookmarks, template)
+
+    def setup_date_format_test(self, date_display_setting: str, web_archive_url: str = ''):
         bookmark = self.setup_bookmark()
         bookmark.date_added = timezone.now() - relativedelta(days=8)
+        bookmark.web_archive_snapshot_url = web_archive_url
         bookmark.save()
         user = self.get_or_create_test_user()
         user.profile.bookmark_date_display = date_display_setting
@@ -35,17 +74,62 @@ class BookmarkListTagTest(TestCase, BookmarkFactoryMixin):
 
     def test_should_respect_absolute_date_setting(self):
         bookmark = self.setup_date_format_test(UserProfile.BOOKMARK_DATE_DISPLAY_ABSOLUTE)
-        html = self.render_template([bookmark])
+        html = self.render_default_template([bookmark])
         formatted_date = formats.date_format(bookmark.date_added, 'SHORT_DATE_FORMAT')
 
-        self.assertInHTML(f'''
-        <span class="text-gray text-sm">{formatted_date}</span>
-        ''', html)
+        self.assertDateLabel(html, formatted_date)
+
+    def test_should_render_web_archive_link_with_absolute_date_setting(self):
+        bookmark = self.setup_date_format_test(UserProfile.BOOKMARK_DATE_DISPLAY_ABSOLUTE,
+                                               'https://web.archive.org/web/20210811214511/https://wanikani.com/')
+        html = self.render_default_template([bookmark])
+        formatted_date = formats.date_format(bookmark.date_added, 'SHORT_DATE_FORMAT')
+
+        self.assertWebArchiveLink(html, formatted_date, bookmark.web_archive_snapshot_url)
 
     def test_should_respect_relative_date_setting(self):
         bookmark = self.setup_date_format_test(UserProfile.BOOKMARK_DATE_DISPLAY_RELATIVE)
-        html = self.render_template([bookmark])
+        html = self.render_default_template([bookmark])
 
-        self.assertInHTML('''
-        <span class="text-gray text-sm">1 week ago</span>
-        ''', html)
+        self.assertDateLabel(html, '1 week ago')
+
+    def test_should_render_web_archive_link_with_relative_date_setting(self):
+        bookmark = self.setup_date_format_test(UserProfile.BOOKMARK_DATE_DISPLAY_RELATIVE,
+                                               'https://web.archive.org/web/20210811214511/https://wanikani.com/')
+        html = self.render_default_template([bookmark])
+
+        self.assertWebArchiveLink(html, '1 week ago', bookmark.web_archive_snapshot_url)
+
+    def test_bookmark_link_target_should_be_blank_by_default(self):
+        bookmark = self.setup_bookmark()
+
+        html = self.render_default_template([bookmark])
+
+        self.assertBookmarksLink(html, bookmark, link_target='_blank')
+
+    def test_bookmark_link_target_should_respect_link_target_parameter(self):
+        bookmark = self.setup_bookmark()
+
+        html = self.render_template_with_link_target([bookmark], '_self')
+
+        self.assertBookmarksLink(html, bookmark, link_target='_self')
+
+    def test_web_archive_link_target_should_be_blank_by_default(self):
+        bookmark = self.setup_bookmark()
+        bookmark.date_added = timezone.now() - relativedelta(days=8)
+        bookmark.web_archive_snapshot_url = 'https://example.com'
+        bookmark.save()
+
+        html = self.render_default_template([bookmark])
+
+        self.assertWebArchiveLink(html, '1 week ago', bookmark.web_archive_snapshot_url, link_target='_blank')
+
+    def test_web_archive_link_target_respect_link_target_parameter(self):
+        bookmark = self.setup_bookmark()
+        bookmark.date_added = timezone.now() - relativedelta(days=8)
+        bookmark.web_archive_snapshot_url = 'https://example.com'
+        bookmark.save()
+
+        html = self.render_template_with_link_target([bookmark], '_self')
+
+        self.assertWebArchiveLink(html, '1 week ago', bookmark.web_archive_snapshot_url, link_target='_self')

bookmarks/tests/test_bookmarks_service.py

@@ -1,11 +1,14 @@
+from unittest.mock import patch
+
 from django.contrib.auth import get_user_model
 from django.test import TestCase
 from django.utils import timezone
 
 from bookmarks.models import Bookmark, Tag
-from bookmarks.services.bookmarks import archive_bookmark, archive_bookmarks, unarchive_bookmark, unarchive_bookmarks, \
-    delete_bookmarks, tag_bookmarks, untag_bookmarks
+from bookmarks.services.bookmarks import create_bookmark, update_bookmark, archive_bookmark, archive_bookmarks, \
+    unarchive_bookmark, unarchive_bookmarks, delete_bookmarks, tag_bookmarks, untag_bookmarks
 from bookmarks.tests.helpers import BookmarkFactoryMixin
+from bookmarks.services import tasks
 
 User = get_user_model()
 
@@ -13,7 +16,30 @@ User = get_user_model()
 class BookmarkServiceTestCase(TestCase, BookmarkFactoryMixin):
 
     def setUp(self) -> None:
-        self.user = User.objects.create_user('testuser', 'test@example.com', 'password123')
+        self.get_or_create_test_user()
+
+    def test_create_should_create_web_archive_snapshot(self):
+        with patch.object(tasks, 'create_web_archive_snapshot') as mock_create_web_archive_snapshot:
+            bookmark_data = Bookmark(url='https://example.com')
+            bookmark = create_bookmark(bookmark_data, 'tag1 tag2', self.user)
+
+            mock_create_web_archive_snapshot.assert_called_once_with(bookmark.id, False)
+
+    def test_update_should_create_web_archive_snapshot_if_url_did_change(self):
+        with patch.object(tasks, 'create_web_archive_snapshot') as mock_create_web_archive_snapshot:
+            bookmark = self.setup_bookmark()
+            bookmark.url = 'https://example.com/updated'
+            update_bookmark(bookmark, 'tag1 tag2', self.user)
+
+            mock_create_web_archive_snapshot.assert_called_once_with(bookmark.id, True)
+
+    def test_update_should_not_create_web_archive_snapshot_if_url_did_not_change(self):
+        with patch.object(tasks, 'create_web_archive_snapshot') as mock_create_web_archive_snapshot:
+            bookmark = self.setup_bookmark()
+            bookmark.title = 'updated title'
+            update_bookmark(bookmark, 'tag1 tag2', self.user)
+
+            mock_create_web_archive_snapshot.assert_not_called()
 
     def test_archive_bookmark(self):
         bookmark = Bookmark(

bookmarks/tests/test_bookmarks_tasks.py

@@ -0,0 +1,154 @@
+from unittest.mock import patch
+
+import waybackpy
+from background_task.models import Task
+from django.contrib.auth.models import User
+from django.test import TestCase, override_settings
+
+from bookmarks.models import Bookmark
+from bookmarks.services.tasks import create_web_archive_snapshot, schedule_bookmarks_without_snapshots
+from bookmarks.tests.helpers import BookmarkFactoryMixin, disable_logging
+
+
+class MockWaybackUrl:
+
+    def __init__(self, archive_url: str):
+        self.archive_url = archive_url
+
+    def save(self):
+        return self
+
+
+class MockWaybackUrlWithSaveError:
+    def save(self):
+        raise NotImplementedError
+
+
+class BookmarkTasksTestCase(TestCase, BookmarkFactoryMixin):
+
+    @disable_logging
+    def run_pending_task(self, task_function):
+        func = getattr(task_function, 'task_function', None)
+        task = Task.objects.all()[0]
+        args, kwargs = task.params()
+        func(*args, **kwargs)
+        task.delete()
+
+    @disable_logging
+    def run_all_pending_tasks(self, task_function):
+        func = getattr(task_function, 'task_function', None)
+        tasks = Task.objects.all()
+
+        for task in tasks:
+            args, kwargs = task.params()
+            func(*args, **kwargs)
+            task.delete()
+
+    def test_create_web_archive_snapshot_should_update_snapshot_url(self):
+        bookmark = self.setup_bookmark()
+
+        with patch.object(waybackpy, 'Url', return_value=MockWaybackUrl('https://example.com')):
+            create_web_archive_snapshot(bookmark.id, False)
+            self.run_pending_task(create_web_archive_snapshot)
+            bookmark.refresh_from_db()
+
+            self.assertEqual(bookmark.web_archive_snapshot_url, 'https://example.com')
+
+    def test_create_web_archive_snapshot_should_handle_missing_bookmark_id(self):
+        with patch.object(waybackpy, 'Url', return_value=MockWaybackUrl('https://example.com')) as mock_wayback_url:
+            create_web_archive_snapshot(123, False)
+            self.run_pending_task(create_web_archive_snapshot)
+
+            mock_wayback_url.assert_not_called()
+
+    def test_create_web_archive_snapshot_should_handle_wayback_save_error(self):
+        bookmark = self.setup_bookmark()
+
+        with patch.object(waybackpy, 'Url',
+                          return_value=MockWaybackUrlWithSaveError()):
+            with self.assertRaises(NotImplementedError):
+                create_web_archive_snapshot(bookmark.id, False)
+                self.run_pending_task(create_web_archive_snapshot)
+
+    def test_create_web_archive_snapshot_should_skip_if_snapshot_exists(self):
+        bookmark = self.setup_bookmark(web_archive_snapshot_url='https://example.com')
+
+        with patch.object(waybackpy, 'Url', return_value=MockWaybackUrl('https://other.com')):
+            create_web_archive_snapshot(bookmark.id, False)
+            self.run_pending_task(create_web_archive_snapshot)
+            bookmark.refresh_from_db()
+
+            self.assertEqual(bookmark.web_archive_snapshot_url, 'https://example.com')
+
+    def test_create_web_archive_snapshot_should_force_update_snapshot(self):
+        bookmark = self.setup_bookmark(web_archive_snapshot_url='https://example.com')
+
+        with patch.object(waybackpy, 'Url', return_value=MockWaybackUrl('https://other.com')):
+            create_web_archive_snapshot(bookmark.id, True)
+            self.run_pending_task(create_web_archive_snapshot)
+            bookmark.refresh_from_db()
+
+            self.assertEqual(bookmark.web_archive_snapshot_url, 'https://other.com')
+
+    @override_settings(LD_DISABLE_BACKGROUND_TASKS=True)
+    def test_create_web_archive_snapshot_should_not_run_when_background_tasks_are_disabled(self):
+        bookmark = self.setup_bookmark()
+        create_web_archive_snapshot(bookmark.id, False)
+
+        self.assertEqual(Task.objects.count(), 0)
+
+    def test_schedule_bookmarks_without_snapshots_should_create_snapshot_task_for_all_bookmarks_without_snapshot(self):
+        user = self.get_or_create_test_user()
+        self.setup_bookmark()
+        self.setup_bookmark()
+        self.setup_bookmark()
+
+        with patch.object(waybackpy, 'Url', return_value=MockWaybackUrl('https://example.com')):
+            schedule_bookmarks_without_snapshots(user.id)
+            self.run_pending_task(schedule_bookmarks_without_snapshots)
+            self.run_all_pending_tasks(create_web_archive_snapshot)
+
+            for bookmark in Bookmark.objects.all():
+                self.assertEqual(bookmark.web_archive_snapshot_url, 'https://example.com')
+
+    def test_schedule_bookmarks_without_snapshots_should_not_update_bookmarks_with_existing_snapshot(self):
+        user = self.get_or_create_test_user()
+        self.setup_bookmark(web_archive_snapshot_url='https://example.com')
+        self.setup_bookmark(web_archive_snapshot_url='https://example.com')
+        self.setup_bookmark(web_archive_snapshot_url='https://example.com')
+
+        with patch.object(waybackpy, 'Url', return_value=MockWaybackUrl('https://other.com')):
+            schedule_bookmarks_without_snapshots(user.id)
+            self.run_pending_task(schedule_bookmarks_without_snapshots)
+            self.run_all_pending_tasks(create_web_archive_snapshot)
+
+            for bookmark in Bookmark.objects.all():
+                self.assertEqual(bookmark.web_archive_snapshot_url, 'https://example.com')
+
+    def test_schedule_bookmarks_without_snapshots_should_only_update_user_owned_bookmarks(self):
+        user = self.get_or_create_test_user()
+        other_user = User.objects.create_user('otheruser', 'otheruser@example.com', 'password123')
+        self.setup_bookmark()
+        self.setup_bookmark()
+        self.setup_bookmark()
+        self.setup_bookmark(user=other_user)
+        self.setup_bookmark(user=other_user)
+        self.setup_bookmark(user=other_user)
+
+        with patch.object(waybackpy, 'Url', return_value=MockWaybackUrl('https://example.com')):
+            schedule_bookmarks_without_snapshots(user.id)
+            self.run_pending_task(schedule_bookmarks_without_snapshots)
+            self.run_all_pending_tasks(create_web_archive_snapshot)
+
+            for bookmark in Bookmark.objects.all().filter(owner=user):
+                self.assertEqual(bookmark.web_archive_snapshot_url, 'https://example.com')
+
+            for bookmark in Bookmark.objects.all().filter(owner=other_user):
+                self.assertEqual(bookmark.web_archive_snapshot_url, '')
+
+    @override_settings(LD_DISABLE_BACKGROUND_TASKS=True)
+    def test_schedule_bookmarks_without_snapshots_should_not_run_when_background_tasks_are_disabled(self):
+        user = self.get_or_create_test_user()
+        schedule_bookmarks_without_snapshots(user.id)
+
+        self.assertEqual(Task.objects.count(), 0)

bookmarks/tests/test_importer.py

@@ -0,0 +1,45 @@
+from unittest.mock import patch
+
+from django.test import TestCase
+
+from bookmarks.services import tasks
+from bookmarks.services.importer import import_netscape_html
+from bookmarks.tests.helpers import BookmarkFactoryMixin, disable_logging
+
+
+class ImporterTestCase(TestCase, BookmarkFactoryMixin):
+
+    def create_import_html(self, bookmark_tags_string: str):
+        return f'''
+<!DOCTYPE NETSCAPE-Bookmark-file-1>
+<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">
+<TITLE>Bookmarks</TITLE>
+<H1>Bookmarks</H1>
+<DL><p>
+{bookmark_tags_string}
+</DL><p>
+        '''
+
+    @disable_logging
+    def test_validate_empty_or_missing_bookmark_url(self):
+        test_html = self.create_import_html(f'''
+            <!-- Empty URL -->
+            <DT><A HREF="" ADD_DATE="1616337559" PRIVATE="0" TOREAD="0" TAGS="tag3">Empty URL</A>
+            <DD>Empty URL
+            <!-- Missing URL -->
+            <DT><A ADD_DATE="1616337559" PRIVATE="0" TOREAD="0" TAGS="tag3">Missing URL</A>
+            <DD>Missing URL
+        ''')
+
+        import_result = import_netscape_html(test_html, self.get_or_create_test_user())
+
+        self.assertEqual(import_result.success, 0)
+
+    def test_schedule_snapshot_creation(self):
+        user = self.get_or_create_test_user()
+        test_html = self.create_import_html('')
+
+        with patch.object(tasks, 'schedule_bookmarks_without_snapshots') as mock_schedule_bookmarks_without_snapshots:
+            import_netscape_html(test_html, user)
+
+            mock_schedule_bookmarks_without_snapshots.assert_called_once_with(user.id)

bookmarks/tests/test_settings_general_view.py

@@ -26,6 +26,7 @@ class SettingsGeneralViewTestCase(TestCase, BookmarkFactoryMixin):
         form_data = {
             'theme': UserProfile.THEME_DARK,
             'bookmark_date_display': UserProfile.BOOKMARK_DATE_DISPLAY_HIDDEN,
+            'bookmark_link_target': UserProfile.BOOKMARK_LINK_TARGET_SELF,
         }
         response = self.client.post(reverse('bookmarks:settings.general'), form_data)
 
@@ -34,3 +35,4 @@ class SettingsGeneralViewTestCase(TestCase, BookmarkFactoryMixin):
         self.assertEqual(response.status_code, 200)
         self.assertEqual(self.user.profile.theme, form_data['theme'])
         self.assertEqual(self.user.profile.bookmark_date_display, form_data['bookmark_date_display'])
+        self.assertEqual(self.user.profile.bookmark_link_target, form_data['bookmark_link_target'])

bookmarks/tests/test_settings_import_view.py

@@ -1,7 +1,7 @@
 from django.test import TestCase
 from django.urls import reverse
 
-from bookmarks.tests.helpers import BookmarkFactoryMixin
+from bookmarks.tests.helpers import BookmarkFactoryMixin, disable_logging
 
 
 class SettingsImportViewTestCase(TestCase, BookmarkFactoryMixin):
@@ -52,6 +52,7 @@ class SettingsImportViewTestCase(TestCase, BookmarkFactoryMixin):
         self.assertNoFormSuccessHint(response)
         self.assertFormErrorHint(response, 'Please select a file to import.')
 
+    @disable_logging
     def test_should_show_hint_if_import_raises_exception(self):
         with open('bookmarks/tests/resources/invalid_import_file.png', 'rb') as import_file:
             response = self.client.post(
@@ -64,6 +65,7 @@ class SettingsImportViewTestCase(TestCase, BookmarkFactoryMixin):
             self.assertNoFormSuccessHint(response)
             self.assertFormErrorHint(response, 'An error occurred during bookmark import.')
 
+    @disable_logging
     def test_should_show_respective_hints_if_not_all_bookmarks_were_imported_successfully(self):
         with open('bookmarks/tests/resources/simple_valid_import_file_with_one_invalid_bookmark.html') as import_file:
             response = self.client.post(

bookmarks/tests/test_signals.py

@@ -0,0 +1,15 @@
+from unittest.mock import patch
+
+from django.test import TestCase
+
+from bookmarks.services import tasks
+from bookmarks.tests.helpers import BookmarkFactoryMixin
+
+
+class SignalsTestCase(TestCase, BookmarkFactoryMixin):
+    def test_login_should_schedule_snapshot_creation(self):
+        user = self.get_or_create_test_user()
+
+        with patch.object(tasks, 'schedule_bookmarks_without_snapshots') as mock_schedule_bookmarks_without_snapshots:
+            self.client.force_login(user)
+            mock_schedule_bookmarks_without_snapshots.assert_called_once_with(user.id)

bookmarks/tests/test_utils.py

@@ -1,9 +1,10 @@
 from unittest.mock import patch
 
+from dateutil.relativedelta import relativedelta
 from django.test import TestCase
 from django.utils import timezone
 
-from bookmarks.utils import humanize_absolute_date, humanize_relative_date
+from bookmarks.utils import humanize_absolute_date, humanize_relative_date, parse_timestamp
 
 
 class UtilsTestCase(TestCase):
@@ -63,3 +64,45 @@ class UtilsTestCase(TestCase):
         # Regression: Test that subsequent calls use current date instead of cached date (#107)
         with patch.object(timezone, 'now', return_value=timezone.datetime(2021, 1, 13)):
             self.assertEqual(humanize_relative_date(timezone.datetime(2021, 1, 13)), 'Today')
+
+    def verify_timestamp(self, date, factor=1):
+        timestamp_string = str(int(date.timestamp() * factor))
+        parsed_date = parse_timestamp(timestamp_string)
+        self.assertEqual(date, parsed_date)
+
+    def test_parse_timestamp_fails_for_invalid_timestamps(self):
+        with self.assertRaises(ValueError):
+            parse_timestamp('invalid')
+
+    def test_parse_timestamp_parses_millisecond_timestamps(self):
+        now = timezone.now().replace(microsecond=0)
+        fifty_years_ago = now - relativedelta(year=50)
+        fifty_years_from_now = now + relativedelta(year=50)
+
+        self.verify_timestamp(now)
+        self.verify_timestamp(fifty_years_ago)
+        self.verify_timestamp(fifty_years_from_now)
+
+    def test_parse_timestamp_parses_microsecond_timestamps(self):
+        now = timezone.now().replace(microsecond=0)
+        fifty_years_ago = now - relativedelta(year=50)
+        fifty_years_from_now = now + relativedelta(year=50)
+
+        self.verify_timestamp(now, 1000)
+        self.verify_timestamp(fifty_years_ago, 1000)
+        self.verify_timestamp(fifty_years_from_now, 1000)
+
+    def test_parse_timestamp_parses_nanosecond_timestamps(self):
+        now = timezone.now().replace(microsecond=0)
+        fifty_years_ago = now - relativedelta(year=50)
+        fifty_years_from_now = now + relativedelta(year=50)
+
+        self.verify_timestamp(now, 1000000)
+        self.verify_timestamp(fifty_years_ago, 1000000)
+        self.verify_timestamp(fifty_years_from_now, 1000000)
+
+    def test_parse_timestamp_fails_for_out_of_range_timestamp(self):
+        now = timezone.now().replace(microsecond=0)
+
+        with self.assertRaises(ValueError):
+            self.verify_timestamp(now, 1000000000)

bookmarks/utils.py

@@ -58,3 +58,40 @@ def humanize_relative_date(value: datetime, now: Optional[datetime] = None):
             return 'Yesterday'
         else:
             return weekday_names[value.isoweekday()]
+
+
+def parse_timestamp(value: str):
+    """
+    Parses a string timestamp into a datetime value
+    First tries to parse the timestamp as milliseconds.
+    If that fails with an error indicating that the timestamp exceeds the maximum,
+    it tries to parse the timestamp as microseconds, and then as nanoseconds
+    :param value:
+    :return:
+    """
+    try:
+        timestamp = int(value)
+    except ValueError:
+        raise ValueError(f'{value} is not a valid timestamp')
+
+    try:
+        return datetime.utcfromtimestamp(timestamp).astimezone()
+    except (OverflowError, ValueError, OSError):
+        pass
+
+    # Value exceeds the max. allowed timestamp
+    # Try parsing as microseconds
+    try:
+        return datetime.utcfromtimestamp(timestamp / 1000).astimezone()
+    except (OverflowError, ValueError, OSError):
+        pass
+
+    # Value exceeds the max. allowed timestamp
+    # Try parsing as nanoseconds
+    try:
+        return datetime.utcfromtimestamp(timestamp / 1000000).astimezone()
+    except (OverflowError, ValueError, OSError):
+        pass
+
+    # Timestamp is out of range
+    raise ValueError(f'{value} exceeds maximum value for a timestamp')

bookmarks/views/bookmarks.py

@@ -40,6 +40,7 @@ def get_bookmark_view_context(request, query_set, tags, base_url):
     paginator = Paginator(query_set, _default_page_size)
     bookmarks = paginator.get_page(page)
     return_url = generate_return_url(base_url, page, query_string)
+    link_target = request.user.profile.bookmark_link_target
 
     if request.GET.get('tag'):
         mod = request.GET.copy()
@@ -51,7 +52,8 @@ def get_bookmark_view_context(request, query_set, tags, base_url):
         'tags': tags,
         'query': query_string if query_string else '',
         'empty': paginator.count == 0,
-        'return_url': return_url
+        'return_url': return_url,
+        'link_target': link_target,
     }
 
 

bookmarks/views/settings.py

@@ -14,6 +14,12 @@ from bookmarks.services import importer
 
 logger = logging.getLogger(__name__)
 
+try:
+    with open("version.txt", "r") as f:
+        app_version = f.read().strip("\n")
+except Exception as exc: 
+    logging.exception(exc)
+    pass
 
 @login_required
 def general(request):
@@ -30,6 +36,7 @@ def general(request):
         'form': form,
         'import_success_message': import_success_message,
         'import_errors_message': import_errors_message,
+        'app_version': app_version
     })
 
 

bootstrap.sh

@@ -12,5 +12,10 @@ python manage.py generate_secret_key
 # Ensure the DB folder is owned by the right user
 chown -R www-data: /etc/linkding/data
 
+# Start background task processor using supervisord, unless explicitly disabled
+if [ "$LD_DISABLE_BACKGROUND_TASKS" != "True" ]; then
+  supervisord -c supervisord.conf
+fi
+
 # Start uwsgi server
 uwsgi uwsgi.ini

docs/Options.md

@@ -25,11 +25,20 @@ All options need to be defined as environment variables in the environment that
 
 ## List of options
 
+### `LD_DISABLE_BACKGROUND_TASKS`
+
+Values: `True`, `False` | Default = `False`
+
+Disables background tasks, such as creating snapshots for bookmarks on the web archive.
+Enabling this flag will prevent the background task processor from starting up, and prevents scheduling tasks.
+This might be useful if you are experiencing performance issues or other problematic behaviour due to background task processing.
+
 ### `LD_DISABLE_URL_VALIDATION`
 
 Values: `True`, `False` | Default = `False`
 
-Completely disables URL validation for bookmarks. This can be useful if you intend to store non fully qualified domain name URLs, such as network paths, or you want to store URLs that use another protocol than `http` or `https`.
+Completely disables URL validation for bookmarks.
+This can be useful if you intend to store non fully qualified domain name URLs, such as network paths, or you want to store URLs that use another protocol than `http` or `https`.
 
 ### `LD_REQUEST_TIMEOUT`
 

package.json

@@ -1,6 +1,6 @@
 {
   "name": "linkding",
-  "version": "1.6.5",
+  "version": "1.8.3",
   "description": "",
   "main": "index.js",
   "scripts": {

requirements.prod.txt

@@ -1,20 +1,26 @@
+asgiref==3.4.1
 beautifulsoup4==4.7.1
 certifi==2019.6.16
-chardet==3.0.4
+charset-normalizer==2.0.4
 confusable-homoglyphs==3.2.0
-Django==2.2.20
+Django==3.2.6
+django-background-tasks==1.2.5
+django-compat==1.0.15
 django-generate-secret-key==1.0.2
-django-picklefield==2.0
-django-registration==3.1.2
-django-sass-processor==0.7.3
-django-widget-tweaks==1.4.5
-djangorestframework==3.11.2
+django-picklefield==3.0.1
+django-registration==3.2
+django-sass-processor==1.0.1
+django-widget-tweaks==1.4.8
+djangorestframework==3.12.4
 idna==2.8
 pyparsing==2.4.7
 python-dateutil==2.8.1
-pytz==2019.1
-requests==2.22.0
+pytz==2021.1
+requests==2.26.0
 soupsieve==1.9.2
-sqlparse==0.3.0
-urllib3==1.25.8
+sqlparse==0.4.1
+supervisor==4.2.2
+typing-extensions==3.10.0.0
+urllib3==1.26.6
 uWSGI==2.0.18
+waybackpy==2.4.3

requirements.txt

@@ -1,27 +1,32 @@
+asgiref==3.4.1
 beautifulsoup4==4.7.1
 certifi==2019.6.16
-chardet==3.0.4
+charset-normalizer==2.0.4
 confusable-homoglyphs==3.2.0
 coverage==5.5
-Django==2.2.20
-django-appconf==1.0.3
-django-compressor==2.3
+Django==3.2.6
+django-appconf==1.0.4
+django-background-tasks==1.2.5
+django-compat==1.0.15
+django-compressor==2.4.1
 django-debug-toolbar==3.2.1
 django-generate-secret-key==1.0.2
-django-picklefield==2.0
-django-registration==3.1.2
-django-sass-processor==0.7.3
-django-widget-tweaks==1.4.5
-djangorestframework==3.11.2
+django-picklefield==3.0.1
+django-registration==3.2
+django-sass-processor==1.0.1
+django-widget-tweaks==1.4.8
+djangorestframework==3.12.4
 idna==2.8
-libsass==0.19.2
+libsass==0.21.0
 pyparsing==2.4.7
 python-dateutil==2.8.1
-pytz==2019.1
+pytz==2021.1
 rcssmin==1.0.6
-requests==2.22.0
+requests==2.26.0
 rjsmin==1.1.0
-six==1.12.0
+six==1.16.0
 soupsieve==1.9.2
-sqlparse==0.3.0
-urllib3==1.25.8
+sqlparse==0.4.1
+typing-extensions==3.10.0.0
+urllib3==1.26.6
+waybackpy==2.4.3

siteroot/settings/base.py

@@ -42,6 +42,7 @@ INSTALLED_APPS = [
     'django_generate_secret_key',
     'rest_framework',
     'rest_framework.authtoken',
+    'background_task',
 ]
 
 MIDDLEWARE = [
@@ -73,6 +74,8 @@ TEMPLATES = [
     },
 ]
 
+DEFAULT_AUTO_FIELD = 'django.db.models.AutoField'
+
 WSGI_APPLICATION = 'siteroot.wsgi.application'
 
 # Database
@@ -164,3 +167,14 @@ ALLOW_REGISTRATION = False
 
 # URL validation flag
 LD_DISABLE_URL_VALIDATION = os.getenv('LD_DISABLE_URL_VALIDATION', False) in (True, 'True', '1')
+
+# Background task enabled setting
+LD_DISABLE_BACKGROUND_TASKS = os.getenv('LD_DISABLE_BACKGROUND_TASKS', False) in (True, 'True', '1')
+
+# django-background-tasks
+MAX_ATTEMPTS = 5
+# How many tasks will run in parallel
+# We want to keep this low to prevent SQLite lock errors and in general not to consume too much resources on smaller
+# specced systems like Raspberries. Should be OK as tasks are not time critical.
+BACKGROUND_TASK_RUN_ASYNC = True
+BACKGROUND_TASK_ASYNC_THREADS = 2

siteroot/settings/dev.py

@@ -43,6 +43,11 @@ LOGGING = {
         'django.db.backends': {
             'level': 'ERROR',  # Set to DEBUG to log all SQL calls
             'handlers': ['console'],
+        },
+        'bookmarks.services.tasks': {  # Log task output
+            'level': 'DEBUG',
+            'handlers': ['console'],
+            'propagate': False,
         }
     }
 }

supervisord.conf

@@ -0,0 +1,10 @@
+[supervisord]
+user=root
+loglevel=info
+
+[program:jobs]
+user=www-data
+command=sh background-tasks-wrapper.sh
+stdout_logfile=/dev/stdout
+stdout_logfile_maxbytes=0
+redirect_stderr=true

version.txt

@@ -1 +1 @@
-1.6.5
+1.8.3