Bump version

The Wayback Machine Save API only allows a limited number of requests within a timespan. This introduces several changes to avoid rate limit errors:
- There will be max. 1 attempt to create a new snapshot
- If a new snapshot could not be created, then attempt to use the latest existing snapshot
- Bulk snapshot updates (bookmark import, load missing snapshots after login) will only attempt to load the latest snapshot instead of creating new ones

.env.sample

@@ -8,7 +8,19 @@ LD_HOST_DATA_DIR=./data
 # Must end with a slash `/`
 LD_CONTEXT_PATH=
 
+# Username of the initial superuser to create, leave empty to not create one
+LD_SUPERUSER_NAME=
+# Password for the initial superuser, leave empty to disable credentials authentication and rely on proxy authentication instead
+LD_SUPERUSER_PASSWORD=
 # Option to disable background tasks
 LD_DISABLE_BACKGROUND_TASKS=False
 # Option to disable URL validation for bookmarks completely
 LD_DISABLE_URL_VALIDATION=False
+# Enables support for authentication proxies such as Authelia
+LD_ENABLE_AUTH_PROXY=False
+# Name of the request header that the auth proxy passes to the application to identify the user
+# See docs/Options.md for more details
+LD_AUTH_PROXY_USERNAME_HEADER=
+# The URL that linkding should redirect to after a logout, when using an auth proxy
+# See docs/Options.md for more details
+LD_AUTH_PROXY_LOGOUT_URL=

.github/workflows/main.yaml

@@ -11,7 +11,7 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@v1
         with:
-          python-version: 3.7
+          python-version: "3.10"
       - name: Set up Node
         uses: actions/setup-node@v2
         with:

CHANGELOG.md

@@ -1,5 +1,26 @@
 # Changelog
 
+## v1.14.0 (14/08/2022)
+
+### What's Changed
+* Add support for context path by @s2marine in https://github.com/sissbruecker/linkding/pull/313
+* Add support for authentication proxies by @sissbruecker in https://github.com/sissbruecker/linkding/pull/321
+* Add bookmark list keyboard navigation by @sissbruecker in https://github.com/sissbruecker/linkding/pull/320
+* Skip updating website metadata on edit unless URL has changed by @sissbruecker in https://github.com/sissbruecker/linkding/pull/318
+* Add simple docs of the new `shared` API parameter by @bachya in https://github.com/sissbruecker/linkding/pull/312
+* Add project linka to community section in README by @cmsax in https://github.com/sissbruecker/linkding/pull/319
+* Order tags in test_should_create_new_bookmark by @RoGryza in https://github.com/sissbruecker/linkding/pull/310
+* Bump django from 3.2.14 to 3.2.15 by @dependabot in https://github.com/sissbruecker/linkding/pull/316
+
+### New Contributors
+* @s2marine made their first contribution in https://github.com/sissbruecker/linkding/pull/313
+* @RoGryza made their first contribution in https://github.com/sissbruecker/linkding/pull/310
+* @cmsax made their first contribution in https://github.com/sissbruecker/linkding/pull/319
+
+**Full Changelog**: https://github.com/sissbruecker/linkding/compare/v1.13.0...v1.14.0
+
+---
+
 ## v1.13.0 (04/08/2022)
 
 ### What's Changed

Dockerfile

@@ -9,7 +9,7 @@ COPY . .
 RUN npm run build
 
 
-FROM python:3.9.6-slim-buster AS python-base
+FROM python:3.10.6-slim-buster AS python-base
 RUN apt-get update && apt-get -y install build-essential
 WORKDIR /etc/linkding
 
@@ -33,7 +33,7 @@ RUN mkdir /opt/venv && \
     /opt/venv/bin/pip install -Ur requirements.txt
 
 
-FROM python:3.9.6-slim-buster as final
+FROM python:3.10.6-slim-buster as final
 RUN apt-get update && apt-get -y install mime-support
 WORKDIR /etc/linkding
 # copy prod dependencies

README.md

@@ -134,12 +134,16 @@ This section lists community projects around using linkding, in alphabetical ord
 - [linkding-cli](https://github.com/bachya/linkding-cli) A command-line interface (CLI) to interact with the linkding REST API. Powered by [aiolinkding](https://github.com/bachya/aiolinkding). By [bachya](https://github.com/bachya)
 - [Open all links bookmarklet](https://gist.github.com/ukcuddlyguy/336dd7339e6d35fc64a75ccfc9323c66) A browser bookmarklet to open all links on the current Linkding page in new tabs. By [ukcuddlyguy](https://github.com/ukcuddlyguy)
 
+## Acknowledgements
+
+JetBrains provides an open-source license of [IntelliJ IDEA](https://www.jetbrains.com/idea/) for the development of linkding.
+
 ## Development
 
-The application is open source, so you are free to modify or contribute. The application is built using the Django web framework. You can get started by checking out the excellent Django docs: https://docs.djangoproject.com/en/3.2/. The `bookmarks` folder contains the actual bookmark application, `siteroot` is the Django root application. Other than that the code should be self-explanatory / standard Django stuff 🙂.
+The application is open source, so you are free to modify or contribute. The application is built using the Django web framework. You can get started by checking out the excellent [Django docs](https://docs.djangoproject.com/en/4.1/). The `bookmarks` folder contains the actual bookmark application, `siteroot` is the Django root application. Other than that the code should be self-explanatory / standard Django stuff 🙂.
 
 ### Prerequisites
-- Python 3
+- Python 3.10
 - Node.js
 
 ### Setup

bookmarks/admin.py

@@ -23,7 +23,25 @@ class AdminBookmark(admin.ModelAdmin):
     search_fields = ('title', 'description', 'website_title', 'website_description', 'url', 'tags__name')
     list_filter = ('owner__username', 'is_archived', 'unread', 'tags',)
     ordering = ('-date_added',)
-    actions = ['archive_selected_bookmarks', 'unarchive_selected_bookmarks', 'mark_as_read', 'mark_as_unread']
+    actions = ['delete_selected_bookmarks', 'archive_selected_bookmarks', 'unarchive_selected_bookmarks', 'mark_as_read', 'mark_as_unread']
+
+    def get_actions(self, request):
+        actions = super().get_actions(request)
+        # Remove default delete action, which gets replaced by delete_selected_bookmarks below
+        # The default action shows a confirmation page which can fail in production when selecting all bookmarks and the
+        # number of objects to delete exceeds the value in DATA_UPLOAD_MAX_NUMBER_FIELDS (1000 by default)
+        del actions['delete_selected']
+        return actions
+
+    def delete_selected_bookmarks(self, request, queryset: QuerySet):
+        bookmarks_count = queryset.count()
+        for bookmark in queryset:
+            bookmark.delete()
+        self.message_user(request, ngettext(
+            '%d bookmark was successfully deleted.',
+            '%d bookmarks were successfully deleted.',
+            bookmarks_count,
+        ) % bookmarks_count, messages.SUCCESS)
 
     def archive_selected_bookmarks(self, request, queryset: QuerySet):
         for bookmark in queryset:

bookmarks/api/serializers.py

@@ -1,4 +1,6 @@
+from django.db.models import prefetch_related_objects
 from rest_framework import serializers
+from rest_framework.serializers import ListSerializer
 
 from bookmarks.models import Bookmark, Tag, build_tag_string
 from bookmarks.services.bookmarks import create_bookmark, update_bookmark
@@ -9,6 +11,14 @@ class TagListField(serializers.ListField):
     child = serializers.CharField()
 
 
+class BookmarkListSerializer(ListSerializer):
+    def to_representation(self, data):
+        # Prefetch nested relations to avoid n+1 queries
+        prefetch_related_objects(data, 'tags')
+
+        return super().to_representation(data)
+
+
 class BookmarkSerializer(serializers.ModelSerializer):
     class Meta:
         model = Bookmark
@@ -32,6 +42,7 @@ class BookmarkSerializer(serializers.ModelSerializer):
             'date_added',
             'date_modified'
         ]
+        list_serializer_class = BookmarkListSerializer
 
     # Override optional char fields to provide default value
     title = serializers.CharField(required=False, allow_blank=True, default='')

bookmarks/management/commands/create_initial_superuser.py

@@ -0,0 +1,37 @@
+import os
+import logging
+
+from django.core.management.base import BaseCommand
+from django.contrib.auth import get_user_model
+
+logger = logging.getLogger(__name__)
+
+
+class Command(BaseCommand):
+    help = "Creates an initial superuser for a deployment using env variables"
+
+    def handle(self, *args, **options):
+        User = get_user_model()
+        superuser_name = os.getenv('LD_SUPERUSER_NAME', None)
+        superuser_password = os.getenv('LD_SUPERUSER_PASSWORD', None)
+
+        # Skip if option is undefined
+        if not superuser_name:
+            logger.info('Skip creating initial superuser, LD_SUPERUSER_NAME option is not defined')
+            return
+
+        # Skip if user already exists
+        user_exists = User.objects.filter(username=superuser_name).exists()
+        if user_exists:
+            logger.info('Skip creating initial superuser, user already exists')
+            return
+
+        user = User(username=superuser_name, is_superuser=True, is_staff=True)
+
+        if superuser_password:
+            user.set_password(superuser_password)
+        else:
+            user.set_unusable_password()
+
+        user.save()
+        logger.info('Created initial superuser')

bookmarks/models.py

@@ -62,11 +62,6 @@ class Bookmark(models.Model):
     owner = models.ForeignKey(get_user_model(), on_delete=models.CASCADE)
     tags = models.ManyToManyField(Tag)
 
-    # Attributes might be calculated in query
-    tag_count = 0  # Projection for number of associated tags
-    tag_string = ''  # Projection for list of tag names, comma-separated
-    tag_projection = False  # Tracks if the above projections were loaded
-
     @property
     def resolved_title(self):
         if self.title:
@@ -82,10 +77,6 @@ class Bookmark(models.Model):
 
     @property
     def tag_names(self):
-        # If tag projections were loaded then avoid querying all tags (=executing further selects)
-        if self.tag_projection:
-            return parse_tag_string(self.tag_string)
-        else:
         return [tag.name for tag in self.tags.all()]
 
     def __str__(self):

bookmarks/queries.py

@@ -1,24 +1,12 @@
 from typing import Optional
 
 from django.contrib.auth.models import User
-from django.db.models import Q, Count, Aggregate, CharField, Value, BooleanField, QuerySet
+from django.db.models import Q, QuerySet
 
 from bookmarks.models import Bookmark, Tag
 from bookmarks.utils import unique
 
 
-class Concat(Aggregate):
-    function = 'GROUP_CONCAT'
-    template = '%(function)s(%(distinct)s%(expressions)s)'
-
-    def __init__(self, expression, distinct=False, **extra):
-        super(Concat, self).__init__(
-            expression,
-            distinct='DISTINCT ' if distinct else '',
-            output_field=CharField(),
-            **extra)
-
-
 def query_bookmarks(user: User, query_string: str) -> QuerySet:
     return _base_bookmarks_query(user, query_string) \
         .filter(is_archived=False)
@@ -36,11 +24,7 @@ def query_shared_bookmarks(user: Optional[User], query_string: str) -> QuerySet:
 
 
 def _base_bookmarks_query(user: Optional[User], query_string: str) -> QuerySet:
-    # Add aggregated tag info to bookmark instances
-    query_set = Bookmark.objects \
-        .annotate(tag_count=Count('tags'),
-                  tag_string=Concat('tags__name'),
-                  tag_projection=Value(True, BooleanField()))
+    query_set = Bookmark.objects
 
     # Filter for user
     if user:

bookmarks/services/tasks.py

@@ -5,8 +5,9 @@ from background_task import background
 from django.conf import settings
 from django.contrib.auth import get_user_model
 from django.contrib.auth.models import User
-from waybackpy.exceptions import WaybackError
+from waybackpy.exceptions import WaybackError, TooManyRequestsError, NoCDXRecordFound
 
+import bookmarks.services.wayback
 from bookmarks.models import Bookmark, UserProfile
 from bookmarks.services.website_loader import DEFAULT_USER_AGENT
 
@@ -26,6 +27,32 @@ def create_web_archive_snapshot(user: User, bookmark: Bookmark, force_update: bo
         _create_web_archive_snapshot_task(bookmark.id, force_update)
 
 
+def _load_newest_snapshot(bookmark: Bookmark):
+    try:
+        logger.info(f'Load existing snapshot for bookmark. url={bookmark.url}')
+        cdx_api = bookmarks.services.wayback.CustomWaybackMachineCDXServerAPI(bookmark.url)
+        existing_snapshot = cdx_api.newest()
+
+        if existing_snapshot:
+            bookmark.web_archive_snapshot_url = existing_snapshot.archive_url
+            bookmark.save()
+            logger.info(f'Using newest snapshot. url={bookmark.url} from={existing_snapshot.datetime_timestamp}')
+
+    except NoCDXRecordFound:
+        logger.info(f'Could not find any snapshots for bookmark. url={bookmark.url}')
+    except WaybackError as error:
+        logger.error(f'Failed to load existing snapshot. url={bookmark.url}', exc_info=error)
+
+
+def _create_snapshot(bookmark: Bookmark):
+    logger.info(f'Create new snapshot for bookmark. url={bookmark.url}...')
+    archive = waybackpy.WaybackMachineSaveAPI(bookmark.url, DEFAULT_USER_AGENT, max_tries=1)
+    archive.save()
+    bookmark.web_archive_snapshot_url = archive.archive_url
+    bookmark.save()
+    logger.info(f'Successfully created new snapshot for bookmark:. url={bookmark.url}')
+
+
 @background()
 def _create_web_archive_snapshot_task(bookmark_id: int, force_update: bool):
     try:
@@ -37,19 +64,31 @@ def _create_web_archive_snapshot_task(bookmark_id: int, force_update: bool):
     if bookmark.web_archive_snapshot_url and not force_update:
         return
 
-    logger.debug(f'Create web archive link for bookmark: {bookmark}...')
-
-    archive = waybackpy.WaybackMachineSaveAPI(bookmark.url, DEFAULT_USER_AGENT)
-
+    # Create new snapshot
     try:
-        archive.save()
+        _create_snapshot(bookmark)
+        return
+    except TooManyRequestsError:
+        logger.error(
+            f'Failed to create snapshot due to rate limiting, trying to load newest snapshot as fallback. url={bookmark.url}')
     except WaybackError as error:
-        logger.exception(f'Error creating web archive link for bookmark: {bookmark}...', exc_info=error)
-        raise
+        logger.error(f'Failed to create snapshot, trying to load newest snapshot as fallback. url={bookmark.url}', exc_info=error)
 
-    bookmark.web_archive_snapshot_url = archive.archive_url
-    bookmark.save()
-    logger.debug(f'Successfully created web archive link for bookmark: {bookmark}...')
+    # Load the newest snapshot as fallback
+    _load_newest_snapshot(bookmark)
+
+
+@background()
+def _load_web_archive_snapshot_task(bookmark_id: int):
+    try:
+        bookmark = Bookmark.objects.get(id=bookmark_id)
+    except Bookmark.DoesNotExist:
+        return
+    # Skip if snapshot exists
+    if bookmark.web_archive_snapshot_url:
+        return
+    # Load the newest snapshot
+    _load_newest_snapshot(bookmark)
 
 
 def schedule_bookmarks_without_snapshots(user: User):
@@ -63,4 +102,6 @@ def _schedule_bookmarks_without_snapshots_task(user_id: int):
     bookmarks_without_snapshots = Bookmark.objects.filter(web_archive_snapshot_url__exact='', owner=user)
 
     for bookmark in bookmarks_without_snapshots:
-        _create_web_archive_snapshot_task(bookmark.id, False)
+        # To prevent rate limit errors from the Wayback API only try to load the latest snapshots instead of creating
+        # new ones when processing bookmarks in bulk
+        _load_web_archive_snapshot_task(bookmark.id)

bookmarks/services/wayback.py

@@ -0,0 +1,40 @@
+import time
+from typing import Dict
+
+import waybackpy
+import waybackpy.utils
+from waybackpy.exceptions import NoCDXRecordFound
+
+
+class CustomWaybackMachineCDXServerAPI(waybackpy.WaybackMachineCDXServerAPI):
+    """
+    Customized WaybackMachineCDXServerAPI to work around some issues with retrieving the newest snapshot.
+    See https://github.com/akamhy/waybackpy/issues/176
+    """
+
+    def newest(self):
+        unix_timestamp = int(time.time())
+        self.closest = waybackpy.utils.unix_timestamp_to_wayback_timestamp(unix_timestamp)
+        self.sort = 'closest'
+        self.limit = -5
+
+        newest_snapshot = None
+        for snapshot in self.snapshots():
+            newest_snapshot = snapshot
+            break
+
+        if not newest_snapshot:
+            raise NoCDXRecordFound(
+                "Wayback Machine's CDX server did not return any records "
+                + "for the query. The URL may not have any archives "
+                + " on the Wayback Machine or the URL may have been recently "
+                + "archived and is still not available on the CDX server."
+            )
+
+        return newest_snapshot
+
+    def add_payload(self, payload: Dict[str, str]) -> None:
+        super().add_payload(payload)
+        # Set fastLatest query param, as we are only using this API to get the latest snapshot and using fastLatest
+        # makes searching for latest snapshots faster
+        payload['fastLatest'] = 'true'

bookmarks/templates/bookmarks/bookmark_list.html

@@ -1,6 +1,6 @@
 {% load shared %}
 {% load pagination %}
-
+{% htmlmin %}
   <ul class="bookmark-list">
     {% for bookmark in bookmarks %}
       <li data-is-bookmark-item>
@@ -23,7 +23,6 @@
             </span>
           {% endif %}
           {% if bookmark.tag_names and bookmark.resolved_description %} | {% endif %}
-
           {% if bookmark.resolved_description %}
             <span>{{ bookmark.resolved_description }}</span>
           {% endif %}
@@ -33,7 +32,8 @@
             <span class="date-label text-gray text-sm">
               {% if bookmark.web_archive_snapshot_url %}
                 <a href="{{ bookmark.web_archive_snapshot_url }}"
-                           title="Show snapshot on the Internet Archive Wayback Machine" target="{{ link_target }}" rel="noopener">
+                   title="Show snapshot on the Internet Archive Wayback Machine" target="{{ link_target }}"
+                   rel="noopener">
               {% endif %}
               <span>{{ bookmark.date_added|humanize_relative_date }}</span>
               {% if bookmark.web_archive_snapshot_url %}
@@ -47,7 +47,8 @@
             <span class="date-label text-gray text-sm">
               {% if bookmark.web_archive_snapshot_url %}
                 <a href="{{ bookmark.web_archive_snapshot_url }}"
-                           title="Show snapshot on the Internet Archive Wayback Machine" target="{{ link_target }}" rel="noopener">
+                   title="Show snapshot on the Internet Archive Wayback Machine" target="{{ link_target }}"
+                   rel="noopener">
               {% endif %}
               <span>{{ bookmark.date_added|humanize_absolute_date }}</span>
               {% if bookmark.web_archive_snapshot_url %}
@@ -63,22 +64,27 @@
                class="btn btn-link btn-sm">Edit</a>
             {% if bookmark.is_archived %}
               <button type="submit" name="unarchive" value="{{ bookmark.id }}"
-                           class="btn btn-link btn-sm">Unarchive</button>
+                      class="btn btn-link btn-sm">Unarchive
+              </button>
             {% else %}
               <button type="submit" name="archive" value="{{ bookmark.id }}"
-                           class="btn btn-link btn-sm">Archive</button>
+                      class="btn btn-link btn-sm">Archive
+              </button>
             {% endif %}
             <button type="submit" name="remove" value="{{ bookmark.id }}"
-                       class="btn btn-link btn-sm btn-confirmation">Remove</button>
+                    class="btn btn-link btn-sm btn-confirmation">Remove
+            </button>
             {% if bookmark.unread %}
               <span class="text-gray text-sm">|</span>
               <button type="submit" name="mark_as_read" value="{{ bookmark.id }}"
-                              class="btn btn-link btn-sm">Mark as read</button>
+                      class="btn btn-link btn-sm">Mark as read
+              </button>
             {% endif %}
           {% else %}
             {# Shared bookmark actions #}
             <span class="text-gray text-sm">Shared by
-                      <a class="text-gray" href="?{% replace_query_param user=bookmark.owner.username %}">{{ bookmark.owner.username }}</a>
+              <a class="text-gray"
+                 href="?{% replace_query_param user=bookmark.owner.username %}">{{ bookmark.owner.username }}</a>
             </span>
           {% endif %}
         </div>
@@ -89,3 +95,4 @@
   <div class="bookmark-pagination">
     {% pagination bookmarks %}
   </div>
+{% endhtmlmin %}

bookmarks/templates/bookmarks/bulk_edit/bar.html

@@ -1,3 +1,5 @@
+{% load shared %}
+{% htmlmin %}
 <div class="bulk-edit-bar">
   <div class="bulk-edit-actions bg-gray">
     <label class="form-checkbox bulk-edit-all-toggle">
@@ -29,3 +31,4 @@
     </button>
   </div>
 </div>
+{% endhtmlmin %}

bookmarks/templates/bookmarks/bulk_edit/toggle.html

@@ -2,7 +2,8 @@
     <span class="btn" title="Bulk edit">
         <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 20 20" fill="currentColor" width="20px"
              height="20px">
-            <path d="M7 3a1 1 0 000 2h6a1 1 0 100-2H7zM4 7a1 1 0 011-1h10a1 1 0 110 2H5a1 1 0 01-1-1zM2 11a2 2 0 012-2h12a2 2 0 012 2v4a2 2 0 01-2 2H4a2 2 0 01-2-2v-4z"/>
+            <path
+                d="M7 3a1 1 0 000 2h6a1 1 0 100-2H7zM4 7a1 1 0 011-1h10a1 1 0 110 2H5a1 1 0 01-1-1zM2 11a2 2 0 012-2h12a2 2 0 012 2v4a2 2 0 01-2 2H4a2 2 0 01-2-2v-4z"/>
         </svg>
     </span>
 </label>

bookmarks/templates/bookmarks/edit.html

@@ -7,7 +7,8 @@
       <div class="content-area-header">
         <h2>Edit bookmark</h2>
       </div>
-            <form action="{% url 'bookmarks:edit' bookmark_id %}?return_url={{ return_url|urlencode }}" method="post" class="col-6 col-md-12" novalidate>
+      <form action="{% url 'bookmarks:edit' bookmark_id %}?return_url={{ return_url|urlencode }}" method="post"
+            class="col-6 col-md-12" novalidate>
         {% bookmark_form form return_url bookmark_id %}
       </form>
     </section>

bookmarks/templates/bookmarks/empty_bookmarks.html

@@ -3,6 +3,7 @@
   <p class="empty-subtitle">
     You can get started by <a href="{% url 'bookmarks:new' %}">adding</a> bookmarks,
     <a href="{% url 'bookmarks:settings.general' %}">importing</a> your existing bookmarks or configuring the
-        <a href="{% url 'bookmarks:settings.integrations' %}">browser extension</a> or the <a href="{% url 'bookmarks:settings.integrations' %}">bookmarklet</a>.
+    <a href="{% url 'bookmarks:settings.integrations' %}">browser extension</a> or the <a
+      href="{% url 'bookmarks:settings.integrations' %}">bookmarklet</a>.
   </p>
 </div>

bookmarks/templates/bookmarks/nav_menu.html

@@ -1,11 +1,16 @@
+{% load shared %}
+{% htmlmin %}
 {# Basic menu list #}
 <div class="hide-md">
   <a href="{% url 'bookmarks:new' %}" class="btn btn-primary mr-2">Add bookmark</a>
   <div class="dropdown">
     <a href="#" class="btn btn-link dropdown-toggle" tabindex="0" style="padding-right: 0.2rem">
       Bookmarks
-          <svg xmlns="http://www.w3.org/2000/svg" class="h-5 w-5" viewBox="0 0 20 20" fill="currentColor" style="height:1rem;width:1rem;vertical-align: text-bottom;">
-            <path fill-rule="evenodd" d="M5.293 7.293a1 1 0 011.414 0L10 10.586l3.293-3.293a1 1 0 111.414 1.414l-4 4a1 1 0 01-1.414 0l-4-4a1 1 0 010-1.414z" clip-rule="evenodd" />
+      <svg xmlns="http://www.w3.org/2000/svg" class="h-5 w-5" viewBox="0 0 20 20" fill="currentColor"
+           style="height:1rem;width:1rem;vertical-align: text-bottom;">
+        <path fill-rule="evenodd"
+              d="M5.293 7.293a1 1 0 011.414 0L10 10.586l3.293-3.293a1 1 0 111.414 1.414l-4 4a1 1 0 01-1.414 0l-4-4a1 1 0 010-1.414z"
+              clip-rule="evenodd"/>
       </svg>
     </a>
     <ul class="menu">
@@ -34,13 +39,15 @@
 {# Menu drop-down for smaller devices #}
 <div class="show-md">
   <a href="{% url 'bookmarks:new' %}" class="btn btn-primary">
-        <svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke="currentColor" style="width: 24px; height: 24px">
+    <svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke="currentColor"
+         style="width: 24px; height: 24px">
       <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M12 6v6m0 0v6m0-6h6m-6 0H6"/>
     </svg>
   </a>
   <div class="dropdown dropdown-right">
     <a href="#" id="mobile-nav-menu-trigger" class="btn btn-link dropdown-toggle" tabindex="0">
-            <svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke="currentColor" style="width: 24px; height: 24px">
+      <svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke="currentColor"
+           style="width: 24px; height: 24px">
         <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 6h16M4 12h16M4 18h16"/>
       </svg>
     </a>
@@ -72,6 +79,7 @@
     </ul>
   </div>
 </div>
+{% endhtmlmin %}
 <script>
   // Hide mobile menu on outside click
   // The Spectre CSS component relies on focus changes to show/hide the dropdown, however mobile browsers like

bookmarks/templates/bookmarks/tag_cloud.html

@@ -1,5 +1,5 @@
 {% load shared %}
-
+{% htmlmin %}
   <div class="tag-cloud">
     {% if has_selected_tags %}
       <p class="selected-tags">
@@ -19,7 +19,8 @@
             {% if forloop.counter == 1 %}
               <a href="?{% append_to_query_param q=tag.name|hash_tag %}"
                  class="mr-2" data-is-tag-item>
-                            <span class="highlight-char">{{ tag.name|first_char }}</span><span>{{ tag.name|remaining_chars:1 }}</span>
+                <span
+                    class="highlight-char">{{ tag.name|first_char }}</span><span>{{ tag.name|remaining_chars:1 }}</span>
               </a>
             {% else %}
               {# Render remaining tags normally #}
@@ -33,3 +34,4 @@
       {% endfor %}
     </div>
   </div>
+{% endhtmlmin %}

bookmarks/templates/settings/general.html

@@ -25,7 +25,8 @@
           <label for="{{ form.bookmark_date_display.id_for_label }}" class="form-label">Bookmark date format</label>
           {{ form.bookmark_date_display|add_class:"form-select col-2 col-sm-12" }}
           <div class="form-input-hint">
-                      Whether to show bookmark dates as relative (how long ago), or as absolute dates. Alternatively the date can be hidden.
+            Whether to show bookmark dates as relative (how long ago), or as absolute dates. Alternatively the date can
+            be hidden.
           </div>
         </div>
         <div class="form-group">
@@ -42,9 +43,11 @@
           <div class="form-input-hint">
             Enabling this feature will automatically create snapshots of bookmarked websites on the <a
               href="https://web.archive.org/" target="_blank" rel="noopener">Internet Archive Wayback
-                      Machine</a>. This allows
-                      to preserve, and later access, the website as it was at the point in time it was bookmarked, in
+            Machine</a>.
+            This allows to preserve, and later access the website as it was at the point in time it was bookmarked, in
             case it goes offline or its content is modified.
+            Please consider donating to the <a href="https://archive.org/donate/index.php" target="_blank"
+                                               rel="noopener">Internet Archive</a> if you make use of this feature.
           </div>
         </div>
         <div class="form-group">

bookmarks/templatetags/shared.py

@@ -1,3 +1,5 @@
+import re
+
 from django import template
 
 from bookmarks import utils
@@ -48,6 +50,7 @@ def remove_from_query_param(context, **kwargs):
 
     return query.urlencode()
 
+
 @register.simple_tag(takes_context=True)
 def replace_query_param(context, **kwargs):
     query = context.request.GET.copy()
@@ -87,3 +90,22 @@ def humanize_relative_date(value):
     if value in (None, ''):
         return ''
     return utils.humanize_relative_date(value)
+
+
+@register.tag
+def htmlmin(parser, token):
+    nodelist = parser.parse(('endhtmlmin',))
+    parser.delete_first_token()
+    return HtmlMinNode(nodelist)
+
+
+class HtmlMinNode(template.Node):
+    def __init__(self, nodelist):
+        self.nodelist = nodelist
+
+    def render(self, context):
+        output = self.nodelist.render(context)
+
+        output = re.sub(r'\s+', ' ', output)
+
+        return output

bookmarks/tests/test_bookmarks_api_performance.py

@@ -0,0 +1,64 @@
+from django.db import connections
+from django.db.utils import DEFAULT_DB_ALIAS
+from django.test.utils import CaptureQueriesContext
+from django.urls import reverse
+from rest_framework import status
+from rest_framework.authtoken.models import Token
+
+from bookmarks.tests.helpers import LinkdingApiTestCase, BookmarkFactoryMixin
+
+
+class BookmarksApiPerformanceTestCase(LinkdingApiTestCase, BookmarkFactoryMixin):
+
+    def setUp(self) -> None:
+        self.api_token = Token.objects.get_or_create(user=self.get_or_create_test_user())[0]
+        self.client.credentials(HTTP_AUTHORIZATION='Token ' + self.api_token.key)
+
+    def get_connection(self):
+        return connections[DEFAULT_DB_ALIAS]
+
+    def test_list_bookmarks_max_queries(self):
+        # set up some bookmarks with associated tags
+        num_initial_bookmarks = 10
+        for index in range(num_initial_bookmarks):
+            self.setup_bookmark(tags=[self.setup_tag()])
+
+        # capture number of queries
+        context = CaptureQueriesContext(self.get_connection())
+        with context:
+            self.get(reverse('bookmarks:bookmark-list'), expected_status_code=status.HTTP_200_OK)
+
+        number_of_queries = context.final_queries
+
+        self.assertLess(number_of_queries, num_initial_bookmarks)
+
+    def test_list_archived_bookmarks_max_queries(self):
+        # set up some bookmarks with associated tags
+        num_initial_bookmarks = 10
+        for index in range(num_initial_bookmarks):
+            self.setup_bookmark(is_archived=True, tags=[self.setup_tag()])
+
+        # capture number of queries
+        context = CaptureQueriesContext(self.get_connection())
+        with context:
+            self.get(reverse('bookmarks:bookmark-archived'), expected_status_code=status.HTTP_200_OK)
+
+        number_of_queries = context.final_queries
+
+        self.assertLess(number_of_queries, num_initial_bookmarks)
+
+    def test_list_shared_bookmarks_max_queries(self):
+        # set up some bookmarks with associated tags
+        share_user = self.setup_user(enable_sharing=True)
+        num_initial_bookmarks = 10
+        for index in range(num_initial_bookmarks):
+            self.setup_bookmark(user=share_user, shared=True, tags=[self.setup_tag()])
+
+        # capture number of queries
+        context = CaptureQueriesContext(self.get_connection())
+        with context:
+            self.get(reverse('bookmarks:bookmark-shared'), expected_status_code=status.HTTP_200_OK)
+
+        number_of_queries = context.final_queries
+
+        self.assertLess(number_of_queries, num_initial_bookmarks)

bookmarks/tests/test_bookmarks_tasks.py

@@ -1,25 +1,51 @@
+import datetime
+from dataclasses import dataclass
 from unittest.mock import patch
 
 import waybackpy
 from background_task.models import Task
 from django.contrib.auth.models import User
 from django.test import TestCase, override_settings
+from waybackpy.exceptions import WaybackError
 
-from bookmarks.models import Bookmark, UserProfile
+import bookmarks.services.wayback
+from bookmarks.models import UserProfile
 from bookmarks.services import tasks
 from bookmarks.tests.helpers import BookmarkFactoryMixin, disable_logging
 
 
 class MockWaybackMachineSaveAPI:
-    def __init__(self, archive_url: str):
+    def __init__(self, archive_url: str = 'https://example.com/created_snapshot', fail_on_save: bool = False):
         self.archive_url = archive_url
+        self.fail_on_save = fail_on_save
 
     def save(self):
+        if self.fail_on_save:
+            raise WaybackError
         return self
 
-class MockWaybackUrlWithSaveError:
-    def save(self):
-        raise NotImplementedError
+
+@dataclass
+class MockCdxSnapshot:
+    archive_url: str
+    datetime_timestamp: datetime.datetime
+
+
+class MockWaybackMachineCDXServerAPI:
+    def __init__(self,
+                 archive_url: str = 'https://example.com/newest_snapshot',
+                 has_no_snapshot=False,
+                 fail_loading_snapshot=False):
+        self.archive_url = archive_url
+        self.has_no_snapshot = has_no_snapshot
+        self.fail_loading_snapshot = fail_loading_snapshot
+
+    def newest(self):
+        if self.has_no_snapshot:
+            return None
+        if self.fail_loading_snapshot:
+            raise WaybackError
+        return MockCdxSnapshot(self.archive_url, datetime.datetime.now())
 
 
 class BookmarkTasksTestCase(TestCase, BookmarkFactoryMixin):
@@ -50,49 +76,130 @@ class BookmarkTasksTestCase(TestCase, BookmarkFactoryMixin):
     def test_create_web_archive_snapshot_should_update_snapshot_url(self):
         bookmark = self.setup_bookmark()
 
-        with patch.object(waybackpy, 'WaybackMachineSaveAPI', return_value=MockWaybackMachineSaveAPI('https://example.com')):
+        with patch.object(waybackpy, 'WaybackMachineSaveAPI', return_value=MockWaybackMachineSaveAPI()):
             tasks.create_web_archive_snapshot(self.get_or_create_test_user(), bookmark, False)
             self.run_pending_task(tasks._create_web_archive_snapshot_task)
             bookmark.refresh_from_db()
 
-            self.assertEqual(bookmark.web_archive_snapshot_url, 'https://example.com')
+            self.assertEqual(bookmark.web_archive_snapshot_url, 'https://example.com/created_snapshot')
 
     def test_create_web_archive_snapshot_should_handle_missing_bookmark_id(self):
-        with patch.object(waybackpy, 'WaybackMachineSaveAPI', return_value=MockWaybackMachineSaveAPI('https://example.com')) as mock_wayback_url:
+        with patch.object(waybackpy, 'WaybackMachineSaveAPI',
+                          return_value=MockWaybackMachineSaveAPI()) as mock_save_api:
             tasks._create_web_archive_snapshot_task(123, False)
             self.run_pending_task(tasks._create_web_archive_snapshot_task)
 
-            mock_wayback_url.assert_not_called()
-
-    def test_create_web_archive_snapshot_should_handle_wayback_save_error(self):
-        bookmark = self.setup_bookmark()
-
-        with patch.object(waybackpy, 'WaybackMachineSaveAPI',
-                          return_value=MockWaybackUrlWithSaveError()):
-            with self.assertRaises(NotImplementedError):
-                tasks.create_web_archive_snapshot(self.get_or_create_test_user(), bookmark, False)
-                self.run_pending_task(tasks._create_web_archive_snapshot_task)
+            mock_save_api.assert_not_called()
 
     def test_create_web_archive_snapshot_should_skip_if_snapshot_exists(self):
         bookmark = self.setup_bookmark(web_archive_snapshot_url='https://example.com')
 
-        with patch.object(waybackpy, 'WaybackMachineSaveAPI', return_value=MockWaybackMachineSaveAPI('https://other.com')):
+        with patch.object(waybackpy, 'WaybackMachineSaveAPI',
+                          return_value=MockWaybackMachineSaveAPI()) as mock_save_api:
             tasks.create_web_archive_snapshot(self.get_or_create_test_user(), bookmark, False)
             self.run_pending_task(tasks._create_web_archive_snapshot_task)
-            bookmark.refresh_from_db()
 
-            self.assertEqual(bookmark.web_archive_snapshot_url, 'https://example.com')
+            mock_save_api.assert_not_called()
 
     def test_create_web_archive_snapshot_should_force_update_snapshot(self):
         bookmark = self.setup_bookmark(web_archive_snapshot_url='https://example.com')
 
-        with patch.object(waybackpy, 'WaybackMachineSaveAPI', return_value=MockWaybackMachineSaveAPI('https://other.com')):
+        with patch.object(waybackpy, 'WaybackMachineSaveAPI',
+                          return_value=MockWaybackMachineSaveAPI('https://other.com')):
             tasks.create_web_archive_snapshot(self.get_or_create_test_user(), bookmark, True)
             self.run_pending_task(tasks._create_web_archive_snapshot_task)
             bookmark.refresh_from_db()
 
             self.assertEqual(bookmark.web_archive_snapshot_url, 'https://other.com')
 
+    def test_create_web_archive_snapshot_should_use_newest_snapshot_as_fallback(self):
+        bookmark = self.setup_bookmark()
+
+        with patch.object(waybackpy, 'WaybackMachineSaveAPI',
+                          return_value=MockWaybackMachineSaveAPI(fail_on_save=True)):
+            with patch.object(bookmarks.services.wayback, 'CustomWaybackMachineCDXServerAPI',
+                              return_value=MockWaybackMachineCDXServerAPI()):
+                tasks.create_web_archive_snapshot(self.get_or_create_test_user(), bookmark, False)
+                self.run_pending_task(tasks._create_web_archive_snapshot_task)
+
+                bookmark.refresh_from_db()
+                self.assertEqual('https://example.com/newest_snapshot', bookmark.web_archive_snapshot_url)
+
+    def test_create_web_archive_snapshot_should_ignore_missing_newest_snapshot(self):
+        bookmark = self.setup_bookmark()
+
+        with patch.object(waybackpy, 'WaybackMachineSaveAPI',
+                          return_value=MockWaybackMachineSaveAPI(fail_on_save=True)):
+            with patch.object(bookmarks.services.wayback, 'CustomWaybackMachineCDXServerAPI',
+                              return_value=MockWaybackMachineCDXServerAPI(has_no_snapshot=True)):
+                tasks.create_web_archive_snapshot(self.get_or_create_test_user(), bookmark, False)
+                self.run_pending_task(tasks._create_web_archive_snapshot_task)
+
+                bookmark.refresh_from_db()
+                self.assertEqual('', bookmark.web_archive_snapshot_url)
+
+    def test_create_web_archive_snapshot_should_ignore_newest_snapshot_errors(self):
+        bookmark = self.setup_bookmark()
+
+        with patch.object(waybackpy, 'WaybackMachineSaveAPI',
+                          return_value=MockWaybackMachineSaveAPI(fail_on_save=True)):
+            with patch.object(bookmarks.services.wayback, 'CustomWaybackMachineCDXServerAPI',
+                              return_value=MockWaybackMachineCDXServerAPI(fail_loading_snapshot=True)):
+                tasks.create_web_archive_snapshot(self.get_or_create_test_user(), bookmark, False)
+                self.run_pending_task(tasks._create_web_archive_snapshot_task)
+
+                bookmark.refresh_from_db()
+                self.assertEqual('', bookmark.web_archive_snapshot_url)
+
+    def test_load_web_archive_snapshot_should_update_snapshot_url(self):
+        bookmark = self.setup_bookmark()
+
+        with patch.object(bookmarks.services.wayback, 'CustomWaybackMachineCDXServerAPI',
+                          return_value=MockWaybackMachineCDXServerAPI()):
+            tasks._load_web_archive_snapshot_task(bookmark.id)
+            self.run_pending_task(tasks._load_web_archive_snapshot_task)
+
+            bookmark.refresh_from_db()
+            self.assertEqual('https://example.com/newest_snapshot', bookmark.web_archive_snapshot_url)
+
+    def test_load_web_archive_snapshot_should_handle_missing_bookmark_id(self):
+        with patch.object(bookmarks.services.wayback, 'CustomWaybackMachineCDXServerAPI',
+                          return_value=MockWaybackMachineCDXServerAPI()) as mock_cdx_api:
+            tasks._load_web_archive_snapshot_task(123)
+            self.run_pending_task(tasks._load_web_archive_snapshot_task)
+
+            mock_cdx_api.assert_not_called()
+
+    def test_load_web_archive_snapshot_should_skip_if_snapshot_exists(self):
+        bookmark = self.setup_bookmark(web_archive_snapshot_url='https://example.com')
+
+        with patch.object(bookmarks.services.wayback, 'CustomWaybackMachineCDXServerAPI',
+                          return_value=MockWaybackMachineCDXServerAPI()) as mock_cdx_api:
+            tasks._load_web_archive_snapshot_task(bookmark.id)
+            self.run_pending_task(tasks._load_web_archive_snapshot_task)
+
+            mock_cdx_api.assert_not_called()
+
+    def test_load_web_archive_snapshot_should_handle_missing_snapshot(self):
+        bookmark = self.setup_bookmark()
+
+        with patch.object(bookmarks.services.wayback, 'CustomWaybackMachineCDXServerAPI',
+                          return_value=MockWaybackMachineCDXServerAPI(has_no_snapshot=True)):
+            tasks._load_web_archive_snapshot_task(bookmark.id)
+            self.run_pending_task(tasks._load_web_archive_snapshot_task)
+
+            self.assertEqual('', bookmark.web_archive_snapshot_url)
+
+    def test_load_web_archive_snapshot_should_handle_wayback_errors(self):
+        bookmark = self.setup_bookmark()
+
+        with patch.object(bookmarks.services.wayback, 'CustomWaybackMachineCDXServerAPI',
+                          return_value=MockWaybackMachineCDXServerAPI(fail_loading_snapshot=True)):
+            tasks._load_web_archive_snapshot_task(bookmark.id)
+            self.run_pending_task(tasks._load_web_archive_snapshot_task)
+
+            self.assertEqual('', bookmark.web_archive_snapshot_url)
+
     @override_settings(LD_DISABLE_BACKGROUND_TASKS=True)
     def test_create_web_archive_snapshot_should_not_run_when_background_tasks_are_disabled(self):
         bookmark = self.setup_bookmark()
@@ -109,33 +216,23 @@ class BookmarkTasksTestCase(TestCase, BookmarkFactoryMixin):
 
         self.assertEqual(Task.objects.count(), 0)
 
-    def test_schedule_bookmarks_without_snapshots_should_create_snapshot_task_for_all_bookmarks_without_snapshot(self):
+    def test_schedule_bookmarks_without_snapshots_should_load_snapshot_for_all_bookmarks_without_snapshot(self):
         user = self.get_or_create_test_user()
         self.setup_bookmark()
         self.setup_bookmark()
         self.setup_bookmark()
+        self.setup_bookmark(web_archive_snapshot_url='https://example.com')
+        self.setup_bookmark(web_archive_snapshot_url='https://example.com')
+        self.setup_bookmark(web_archive_snapshot_url='https://example.com')
 
-        with patch.object(waybackpy, 'WaybackMachineSaveAPI', return_value=MockWaybackMachineSaveAPI('https://example.com')):
         tasks.schedule_bookmarks_without_snapshots(user)
         self.run_pending_task(tasks._schedule_bookmarks_without_snapshots_task)
-            self.run_all_pending_tasks(tasks._create_web_archive_snapshot_task)
 
-            for bookmark in Bookmark.objects.all():
-                self.assertEqual(bookmark.web_archive_snapshot_url, 'https://example.com')
+        task_list = Task.objects.all()
+        self.assertEqual(task_list.count(), 3)
 
-    def test_schedule_bookmarks_without_snapshots_should_not_update_bookmarks_with_existing_snapshot(self):
-        user = self.get_or_create_test_user()
-        self.setup_bookmark(web_archive_snapshot_url='https://example.com')
-        self.setup_bookmark(web_archive_snapshot_url='https://example.com')
-        self.setup_bookmark(web_archive_snapshot_url='https://example.com')
-
-        with patch.object(waybackpy, 'WaybackMachineSaveAPI', return_value=MockWaybackMachineSaveAPI('https://other.com')):
-            tasks.schedule_bookmarks_without_snapshots(user)
-            self.run_pending_task(tasks._schedule_bookmarks_without_snapshots_task)
-            self.run_all_pending_tasks(tasks._create_web_archive_snapshot_task)
-
-            for bookmark in Bookmark.objects.all():
-                self.assertEqual(bookmark.web_archive_snapshot_url, 'https://example.com')
+        for task in task_list:
+            self.assertEqual(task.task_name, 'bookmarks.services.tasks._load_web_archive_snapshot_task')
 
     def test_schedule_bookmarks_without_snapshots_should_only_update_user_owned_bookmarks(self):
         user = self.get_or_create_test_user()
@@ -147,16 +244,11 @@ class BookmarkTasksTestCase(TestCase, BookmarkFactoryMixin):
         self.setup_bookmark(user=other_user)
         self.setup_bookmark(user=other_user)
 
-        with patch.object(waybackpy, 'WaybackMachineSaveAPI', return_value=MockWaybackMachineSaveAPI('https://example.com')):
         tasks.schedule_bookmarks_without_snapshots(user)
         self.run_pending_task(tasks._schedule_bookmarks_without_snapshots_task)
-            self.run_all_pending_tasks(tasks._create_web_archive_snapshot_task)
 
-            for bookmark in Bookmark.objects.all().filter(owner=user):
-                self.assertEqual(bookmark.web_archive_snapshot_url, 'https://example.com')
-
-            for bookmark in Bookmark.objects.all().filter(owner=other_user):
-                self.assertEqual(bookmark.web_archive_snapshot_url, '')
+        task_list = Task.objects.all()
+        self.assertEqual(task_list.count(), 3)
 
     @override_settings(LD_DISABLE_BACKGROUND_TASKS=True)
     def test_schedule_bookmarks_without_snapshots_should_not_run_when_background_tasks_are_disabled(self):

bookmarks/tests/test_create_initial_superuser_command.py

@@ -0,0 +1,45 @@
+import os
+from unittest import mock
+
+from django.test import TestCase
+
+from bookmarks.models import User
+from bookmarks.management.commands.create_initial_superuser import Command
+
+
+class TestCreateInitialSuperuserCommand(TestCase):
+
+    @mock.patch.dict(os.environ, {'LD_SUPERUSER_NAME': 'john', 'LD_SUPERUSER_PASSWORD': 'password123'})
+    def test_create_with_password(self):
+        Command().handle()
+
+        self.assertEqual(1, User.objects.count())
+
+        user = User.objects.first()
+        self.assertEqual('john', user.username)
+        self.assertTrue(user.has_usable_password())
+        self.assertTrue(user.check_password('password123'))
+
+    @mock.patch.dict(os.environ, {'LD_SUPERUSER_NAME': 'john'})
+    def test_create_without_password(self):
+        Command().handle()
+
+        self.assertEqual(1, User.objects.count())
+
+        user = User.objects.first()
+        self.assertEqual('john', user.username)
+        self.assertFalse(user.has_usable_password())
+
+    def test_create_without_options(self):
+        Command().handle()
+
+        self.assertEqual(0, User.objects.count())
+
+    @mock.patch.dict(os.environ, {'LD_SUPERUSER_NAME': 'john', 'LD_SUPERUSER_PASSWORD': 'password123'})
+    def test_create_multiple_times(self):
+        Command().handle()
+        Command().handle()
+        Command().handle()
+
+        self.assertEqual(1, User.objects.count())
+

bookmarks/tests/test_exporter_performance.py

@@ -0,0 +1,32 @@
+from django.db import connections
+from django.db.utils import DEFAULT_DB_ALIAS
+from django.test import TestCase
+from django.test.utils import CaptureQueriesContext
+from django.urls import reverse
+
+from bookmarks.tests.helpers import BookmarkFactoryMixin
+
+
+class ExporterPerformanceTestCase(TestCase, BookmarkFactoryMixin):
+
+    def setUp(self) -> None:
+        user = self.get_or_create_test_user()
+        self.client.force_login(user)
+
+    def get_connection(self):
+        return connections[DEFAULT_DB_ALIAS]
+
+    def test_export_max_queries(self):
+        # set up some bookmarks with associated tags
+        num_initial_bookmarks = 10
+        for index in range(num_initial_bookmarks):
+            self.setup_bookmark(tags=[self.setup_tag()])
+
+        # capture number of queries
+        context = CaptureQueriesContext(self.get_connection())
+        with context:
+            self.client.get(reverse('bookmarks:settings.export'),follow=True)
+
+        number_of_queries = context.final_queries
+
+        self.assertLess(number_of_queries, num_initial_bookmarks)

bookmarks/tests/test_feeds.py

@@ -35,13 +35,13 @@ class FeedsTestCase(TestCase, BookmarkFactoryMixin):
         self.assertContains(response, '<title>All bookmarks</title>')
         self.assertContains(response, '<description>All bookmarks</description>')
         self.assertContains(response, f'<link>http://testserver{feed_url}</link>')
-        self.assertContains(response, f'<atom:link href="http://testserver{feed_url}" rel="self"></atom:link>')
+        self.assertContains(response, f'<atom:link href="http://testserver{feed_url}" rel="self"/>')
 
     def test_all_returns_all_unarchived_bookmarks(self):
         bookmarks = [
-            self.setup_bookmark(),
-            self.setup_bookmark(),
-            self.setup_bookmark(unread=True),
+            self.setup_bookmark(description='test description'),
+            self.setup_bookmark(website_description='test website description'),
+            self.setup_bookmark(unread=True, description='test description'),
         ]
         self.setup_bookmark(is_archived=True)
         self.setup_bookmark(is_archived=True)
@@ -117,7 +117,7 @@ class FeedsTestCase(TestCase, BookmarkFactoryMixin):
         self.assertContains(response, '<title>Unread bookmarks</title>')
         self.assertContains(response, '<description>All unread bookmarks</description>')
         self.assertContains(response, f'<link>http://testserver{feed_url}</link>')
-        self.assertContains(response, f'<atom:link href="http://testserver{feed_url}" rel="self"></atom:link>')
+        self.assertContains(response, f'<atom:link href="http://testserver{feed_url}" rel="self"/>')
 
     def test_unread_returns_unread_and_unarchived_bookmarks(self):
         self.setup_bookmark(unread=False)
@@ -128,9 +128,9 @@ class FeedsTestCase(TestCase, BookmarkFactoryMixin):
         self.setup_bookmark(unread=False, is_archived=True)
 
         unread_bookmarks = [
-            self.setup_bookmark(unread=True),
-            self.setup_bookmark(unread=True),
-            self.setup_bookmark(unread=True),
+            self.setup_bookmark(unread=True, description='test description'),
+            self.setup_bookmark(unread=True, website_description='test website description'),
+            self.setup_bookmark(unread=True, description='test description'),
         ]
 
         response = self.client.get(reverse('bookmarks:feeds.unread', args=[self.token.key]))

bookmarks/tests/test_feeds_performance.py

@@ -0,0 +1,35 @@
+from django.db import connections
+from django.db.utils import DEFAULT_DB_ALIAS
+from django.test import TestCase
+from django.test.utils import CaptureQueriesContext
+from django.urls import reverse
+
+from bookmarks.models import FeedToken
+from bookmarks.tests.helpers import BookmarkFactoryMixin
+
+
+class FeedsPerformanceTestCase(TestCase, BookmarkFactoryMixin):
+
+    def setUp(self) -> None:
+        user = self.get_or_create_test_user()
+        self.client.force_login(user)
+        self.token = FeedToken.objects.get_or_create(user=user)[0]
+
+    def get_connection(self):
+        return connections[DEFAULT_DB_ALIAS]
+
+    def test_all_max_queries(self):
+        # set up some bookmarks with associated tags
+        num_initial_bookmarks = 10
+        for index in range(num_initial_bookmarks):
+            self.setup_bookmark(tags=[self.setup_tag()])
+
+        # capture number of queries
+        context = CaptureQueriesContext(self.get_connection())
+        with context:
+            feed_url = reverse('bookmarks:feeds.all', args=[self.token.key])
+            self.client.get(feed_url)
+
+        number_of_queries = context.final_queries
+
+        self.assertLess(number_of_queries, num_initial_bookmarks)

bookmarks/tests/test_queries.py

@@ -270,25 +270,6 @@ class QueriesTestCase(TestCase, BookmarkFactoryMixin):
 
         self.assertQueryResult(query, [owned_bookmarks])
 
-    def test_query_bookmarks_should_use_tag_projection(self):
-        self.setup_bookmark_search_data()
-
-        # Test projection on bookmarks with tags
-        query = queries.query_bookmarks(self.user, '#tag1 #tag2')
-
-        for bookmark in query:
-            self.assertEqual(bookmark.tag_count, 2)
-            self.assertEqual(bookmark.tag_string, 'tag1,tag2')
-            self.assertTrue(bookmark.tag_projection)
-
-        # Test projection on bookmarks without tags
-        query = queries.query_bookmarks(self.user, 'term2')
-
-        for bookmark in query:
-            self.assertEqual(bookmark.tag_count, 0)
-            self.assertEqual(bookmark.tag_string, None)
-            self.assertTrue(bookmark.tag_projection)
-
     def test_query_bookmarks_untagged_should_return_untagged_bookmarks_only(self):
         tag = self.setup_tag()
         untagged_bookmark = self.setup_bookmark()

bookmarks/urls.py

@@ -1,4 +1,4 @@
-from django.conf.urls import url
+from django.urls import re_path
 from django.urls import path, include
 from django.views.generic import RedirectView
 
@@ -9,7 +9,7 @@ from bookmarks.feeds import AllBookmarksFeed, UnreadBookmarksFeed
 app_name = 'bookmarks'
 urlpatterns = [
     # Redirect root to bookmarks index
-    url(r'^$', RedirectView.as_view(pattern_name='bookmarks:index', permanent=False)),
+    re_path(r'^$', RedirectView.as_view(pattern_name='bookmarks:index', permanent=False)),
     # Bookmarks
     path('bookmarks', views.bookmarks.index, name='index'),
     path('bookmarks/archived', views.bookmarks.archived, name='archived'),

bookmarks/views/bookmarks.py

@@ -73,8 +73,8 @@ def get_bookmark_view_context(request: WSGIRequest,
     paginator = Paginator(query_set, _default_page_size)
     bookmarks = paginator.get_page(page)
     selected_tags = _get_selected_tags(tags, filters.query)
-    # Prefetch owner relation, this avoids n+1 queries when using the owner in templates
-    prefetch_related_objects(bookmarks.object_list, 'owner')
+    # Prefetch related objects, this avoids n+1 queries when accessing fields in templates
+    prefetch_related_objects(bookmarks.object_list, 'owner', 'tags')
     return_url = generate_return_url(base_url, page, filters)
     link_target = request.user.profile.bookmark_link_target
 

bookmarks/views/settings.py

@@ -5,6 +5,7 @@ from functools import lru_cache
 import requests
 from django.contrib import messages
 from django.contrib.auth.decorators import login_required
+from django.db.models import prefetch_related_objects
 from django.http import HttpResponseRedirect, HttpResponse
 from django.shortcuts import render
 from django.urls import reverse
@@ -114,7 +115,9 @@ def bookmark_import(request):
 def bookmark_export(request):
     # noinspection PyBroadException
     try:
-        bookmarks = query_bookmarks(request.user, '')
+        bookmarks = list(query_bookmarks(request.user, ''))
+        # Prefetch tags to prevent n+1 queries
+        prefetch_related_objects(bookmarks, 'tags')
         file_content = exporter.export_netscape_html(bookmarks)
 
         response = HttpResponse(content_type='text/plain; charset=UTF-8')

bootstrap.sh

@@ -10,6 +10,8 @@ mkdir -p data
 python manage.py migrate
 # Generate secret key file if it does not exist
 python manage.py generate_secret_key
+# Create initial superuser if defined in options / environment variables
+python manage.py create_initial_superuser
 
 # Ensure the DB folder is owned by the right user
 chown -R www-data: /etc/linkding/data

docs/Options.md

@@ -25,6 +25,22 @@ All options need to be defined as environment variables in the environment that
 
 ## List of options
 
+### `LD_SUPERUSER_NAME`
+
+Values: `String` | Default = None
+
+When set, creates an initial superuser with the specified username when starting the container.
+Does nothing if the user already exists.
+
+See [`LD_SUPERUSER_PASSWORD`](#ld_superuser_password) on how to configure the respective password.
+
+### `LD_SUPERUSER_PASSWORD`
+
+Values: `String` | Default = None
+
+The password for the initial superuser.
+When left undefined, the superuser will be created without a usable password, which means the user can not authenticate using credentials / through the login form, and can only be authenticated using proxy authentication (see [`LD_ENABLE_AUTH_PROXY`](#ld_enable_auth_proxy)).
+
 ### `LD_DISABLE_BACKGROUND_TASKS`
 
 Values: `True`, `False` | Default = `False`

docs/how-to.md

@@ -22,17 +22,17 @@ For more info see here: https://paul.kinlan.me/use-bookmarklets-on-chrome-on-and
 
 ## Using HTTP Shortcuts app on Android
 
-**Note** This allows you to share URL from any app to bookmark it to linkding
+**Note** This allows you to share URL from any app to tag and bookmark it to linkding
 
 - Install HTTP Shortcuts from [Play Store](https://play.google.com/store/apps/details?id=ch.rmy.android.http_shortcuts) or [F-Droid](https://f-droid.org/en/packages/ch.rmy.android.http_shortcuts/).
 
-- Download [linkding_shortcut.json](/docs/linkding_shortcut.json) from this repository.
+- Copy the raw URL of [linkding_shortcut.json](/docs/linkding_shortcut.json) in this repository.
 
-- Open HTTP Shortcuts, tap the 3-dot-button at the top-right corner, tap `Import/Export`, then tap `Import from file`.
+- Open HTTP Shortcuts, tap the 3-dot-button at the top-right corner, tap `Import/Export`, then tap `Import from URL`.
 
-- Select the json file you downloaded earlier, go back, tap the 3-dot-button again, then tap `Variables`.
+- Paste the URL you copied earlier, tap OK, go back, tap the 3-dot-button again, then tap `Variables`.
 
-- Edit the `values` of `linkding_instance`, `linkding_tag` and `linkding_api_token`.
+- Edit the `values` of `linkding_instance` and `linkding_api_token`.
 
 Try using share button on an app, a new item `Send to...` should appear on the share sheet. You can also manually share by tapping the shortcut inside the HTTP Shortcuts app itself.
 

docs/linkding_shortcut.json

@@ -5,12 +5,12 @@
       "name": "Shortcuts",
       "shortcuts": [
         {
-          "bodyContent": "{ \"url\": \"{{b2953f61-b302-4c79-b90d-39858a06d9a6}}\", \"tag_names\": [ \"{{c360f61f-ce17-47b4-bea3-1d8c3913ca52}}\" ] }",
+          "bodyContent": "{ \"url\": \"{{b2953f61-b302-4c79-b90d-39858a06d9a6}}\", \"tag_names\": [ \"{{7871474b-e325-4ca0-a142-a5ef5d3f7ed8}}\" ] }",
           "contentType": "application/json",
           "description": "Bookmark to linkding",
           "headers": [
             {
-              "id": "d235f7b4-fce2-41f4-a00f-72d5fde9e4b9",
+              "id": "fd6306d7-e09d-4c14-a538-3fc258460028",
               "key": "Authorization",
               "value": "Token {{6a739a16-d16d-4a06-93a5-3457da3c3d20}}"
             }
@@ -23,7 +23,6 @@
           "quickSettingsTileShortcut": true,
           "responseHandling": {
             "failureOutput": "simple",
-            "id": "61fa9fc3-8b7a-47ce-b43c-f24618a65e1e",
             "uiType": "toast"
           },
           "url": "{{ea2db14b-b9ca-45d8-8555-403271a38f5a}}/api/bookmarks/"
@@ -44,16 +43,18 @@
       "title": "Enter URL",
       "type": "text"
     },
-    {
-      "id": "c360f61f-ce17-47b4-bea3-1d8c3913ca52",
-      "key": "linkding_tag",
-      "value": "single-tag"
-    },
     {
       "id": "6a739a16-d16d-4a06-93a5-3457da3c3d20",
       "key": "linkding_api_token",
       "value": "your_token_from_integrations_tab"
+    },
+    {
+      "id": "7871474b-e325-4ca0-a142-a5ef5d3f7ed8",
+      "key": "linkding_custom_tag",
+      "message": "Enter one or more comma separated tags",
+      "title": "Tag",
+      "type": "text"
     }
   ],
-  "version": 45
+  "version": 53
 }

package.json

@@ -1,6 +1,6 @@
 {
   "name": "linkding",
-  "version": "1.14.0",
+  "version": "1.15.0",
   "description": "",
   "main": "index.js",
   "scripts": {

requirements.prod.txt

@@ -1,25 +1,24 @@
-asgiref==3.4.1
-beautifulsoup4==4.7.1
-certifi==2019.6.16
-charset-normalizer==2.0.4
+asgiref==3.5.2
+beautifulsoup4==4.11.1
+certifi==2022.6.15
+charset-normalizer==2.1.1
+click==8.1.3
 confusable-homoglyphs==3.2.0
-Django==3.2.15
-django-background-tasks==1.2.5
-django-compat==1.0.15
+Django==4.1
 django-generate-secret-key==1.0.2
-django-picklefield==3.0.1
-django-registration==3.2
-django-sass-processor==1.0.1
-django-widget-tweaks==1.4.8
-djangorestframework==3.12.4
-idna==2.8
-python-dateutil==2.8.1
-pytz==2021.1
-requests==2.26.0
-soupsieve==1.9.2
+django-registration==3.3
+django-sass-processor==1.2.1
+django-widget-tweaks==1.4.12
+django4-background-tasks==1.2.7
+djangorestframework==3.13.1
+idna==3.3
+python-dateutil==2.8.2
+pytz==2022.2.1
+requests==2.28.1
+soupsieve==2.3.2.post1
 sqlparse==0.4.2
-supervisor==4.2.2
+supervisor==4.2.4
 typing-extensions==3.10.0.0
-urllib3==1.26.6
-uWSGI==2.0.18
+urllib3==1.26.11
+uWSGI==2.0.20
 waybackpy==3.0.6

requirements.txt

@@ -1,31 +1,30 @@
-asgiref==3.4.1
-beautifulsoup4==4.7.1
-certifi==2019.6.16
-charset-normalizer==2.0.4
+asgiref==3.5.2
+beautifulsoup4==4.11.1
+certifi==2022.6.15
+charset-normalizer==2.1.1
+click==8.1.3
 confusable-homoglyphs==3.2.0
 coverage==5.5
-Django==3.2.15
-django-appconf==1.0.4
-django-background-tasks==1.2.5
-django-compat==1.0.15
-django-compressor==2.4.1
-django-debug-toolbar==3.2.1
+Django==4.1
+django-appconf==1.0.5
+django-compressor==4.1
+django-debug-toolbar==3.6.0
 django-generate-secret-key==1.0.2
-django-picklefield==3.0.1
-django-registration==3.2
-django-sass-processor==1.0.1
-django-widget-tweaks==1.4.8
-djangorestframework==3.12.4
-idna==2.8
+django-registration==3.3
+django-sass-processor==1.2.1
+django-widget-tweaks==1.4.12
+django4-background-tasks==1.2.7
+djangorestframework==3.13.1
+idna==3.3
 libsass==0.21.0
-python-dateutil==2.8.1
-pytz==2021.1
-rcssmin==1.0.6
-requests==2.26.0
-rjsmin==1.1.0
+python-dateutil==2.8.2
+pytz==2022.2.1
+rcssmin==1.1.0
+requests==2.28.1
+rjsmin==1.2.0
 six==1.16.0
-soupsieve==1.9.2
+soupsieve==2.3.2.post1
 sqlparse==0.4.2
 typing-extensions==3.10.0.0
-urllib3==1.26.6
+urllib3==1.26.11
 waybackpy==3.0.6

siteroot/settings/prod.py

@@ -28,6 +28,35 @@ if host_name:
 else:
     ALLOWED_HOSTS = ['*']
 
+# Logging
+LOGGING = {
+    'version': 1,
+    'disable_existing_loggers': False,
+    'formatters': {
+        'simple': {
+            'format': '{asctime} {levelname} {message}',
+            'style': '{',
+        },
+    },
+    'handlers': {
+        'console': {
+            'class': 'logging.StreamHandler',
+            'formatter': 'simple'
+        }
+    },
+    'root': {
+        'handlers': ['console'],
+        'level': 'WARN',
+    },
+    'loggers': {
+        'bookmarks': {
+            'level': 'INFO',
+            'handlers': ['console'],
+            'propagate': False,
+        }
+    }
+}
+
 # Import custom settings
 # noinspection PyUnresolvedReferences
 from .custom import *

supervisord.conf

@@ -5,6 +5,7 @@ loglevel=info
 [program:jobs]
 user=www-data
 command=sh background-tasks-wrapper.sh
-stdout_logfile=/dev/stdout
-stdout_logfile_maxbytes=0
+stdout_logfile=background_tasks.log
+stdout_logfile_maxbytes=10MB
+stdout_logfile_backups=5
 redirect_stderr=true

version.txt

@@ -1 +1 @@
-1.14.0
+1.15.0