Bump version

Co-authored-by: Sascha Ißbrücker <sascha.issbruecker@gmail.com>

CHANGELOG.md

@@ -1,5 +1,40 @@
 # Changelog
 
+## v1.10.0 (21/05/2022)
+### What's Changed
+* Add to managed hosting options by @m3nu in https://github.com/sissbruecker/linkding/pull/253
+* Add community reference to aiolinkding by @bachya in https://github.com/sissbruecker/linkding/pull/259
+* Improve import performance by @sissbruecker in https://github.com/sissbruecker/linkding/pull/261
+* Update how-to.md to fix unclear/paraphrased Safari action in IOS Shortcuts by @feoh in https://github.com/sissbruecker/linkding/pull/260
+* Allow searching for untagged bookmarks by @sissbruecker in https://github.com/sissbruecker/linkding/pull/226
+
+### New Contributors
+* @m3nu made their first contribution in https://github.com/sissbruecker/linkding/pull/253
+* @bachya made their first contribution in https://github.com/sissbruecker/linkding/pull/259
+* @feoh made their first contribution in https://github.com/sissbruecker/linkding/pull/260
+
+**Full Changelog**: https://github.com/sissbruecker/linkding/compare/v1.9.0...v1.10.0
+
+---
+
+## v1.9.0 (14/05/2022)
+### What's Changed
+* Scroll menu items into view when using keyboard by @sissbruecker in https://github.com/sissbruecker/linkding/pull/248
+* Add whitespace after auto-completed tag by @sissbruecker in https://github.com/sissbruecker/linkding/pull/249
+* Bump django from 3.2.12 to 3.2.13 by @dependabot in https://github.com/sissbruecker/linkding/pull/244
+* Add community helm chart reference to readme by @pascaliske in https://github.com/sissbruecker/linkding/pull/242
+* Feature: Shortcut key for new bookmark by @rithask in https://github.com/sissbruecker/linkding/pull/241
+* Clarify archive.org feature by @clach04 in https://github.com/sissbruecker/linkding/pull/229
+* Make Internet Archive integration opt-in by @sissbruecker in https://github.com/sissbruecker/linkding/pull/250
+
+### New Contributors
+* @pascaliske made their first contribution in https://github.com/sissbruecker/linkding/pull/242
+* @rithask made their first contribution in https://github.com/sissbruecker/linkding/pull/241
+* @clach04 made their first contribution in https://github.com/sissbruecker/linkding/pull/229
+
+**Full Changelog**: https://github.com/sissbruecker/linkding/compare/v1.8.8...v1.9.0
+---
+
 ## v1.8.8 (27/03/2022)
 - [**bug**] Prevent bookmark actions through get requests
 - [**bug**] Prevent external redirects
@@ -156,21 +191,4 @@
 ---
 
 ## v1.1.1 (01/01/2021)
-- [**enhancement**] Add docker-compose support [#54](https://github.com/sissbruecker/linkding/pull/54)
+- [**enhancement**] Add docker-compose support [#54](https://github.com/sissbruecker/linkding/pull/54)
----
-
-## v1.1.0 (31/12/2020)
-- [**enhancement**] Search autocomplete [#52](https://github.com/sissbruecker/linkding/issues/52)
-- [**enhancement**] Improve Netscape bookmarks file parsing [#50](https://github.com/sissbruecker/linkding/issues/50)
----
-
-## v1.0.0 (31/12/2020)
-- [**bug**] Import does not import bookmark descriptions [#47](https://github.com/sissbruecker/linkding/issues/47)
-- [**enhancement**] Enhancement: return to same page we were on after editing a bookmark [#26](https://github.com/sissbruecker/linkding/issues/26)
-- [**bug**] Increase limit on bookmark URL length [#25](https://github.com/sissbruecker/linkding/issues/25)
-- [**enhancement**] API for app development [#24](https://github.com/sissbruecker/linkding/issues/24)
-- [**enhancement**] Enhancement: detect duplicates at entry time [#23](https://github.com/sissbruecker/linkding/issues/23)
-- [**bug**] Error importing bookmarks [#18](https://github.com/sissbruecker/linkding/issues/18)
-- [**enhancement**] Enhancement: better administration page [#4](https://github.com/sissbruecker/linkding/issues/4)
-- [**enhancement**] Bug: Navigation bar active link stays on add bookmark [#3](https://github.com/sissbruecker/linkding/issues/3)
-- [**bug**] CSS Stylesheet presented as text/plain [#2](https://github.com/sissbruecker/linkding/issues/2)

README.md

@@ -1,12 +1,32 @@
-#  linkding
+<div align="center">
+    <br>
+    <a href="https://github.com/sissbruecker/linkding">
+        <img src="docs/header.svg" height="50">
+    </a>
+    <br>
+</div>
 
-*linkding* is a simple bookmark service that you can host yourself.
+## Overview
-It's designed be to be minimal, fast and easy to set up using Docker. 
+- [Introduction](#introduction)
+- [Installation](#installation)
+    - [Using Docker](#using-docker)
+    - [Using Docker Compose](#using-docker-compose)
+    - [User Setup](#user-setup)
+    - [Managed Hosting Options](#managed-hosting-options)
+- [Documentation](#documentation)
+- [Browser Extension](#browser-extension)
+- [Community](#community)
+- [Development](#development)
+
+##  Introduction
+
+linkding is a simple bookmark service that you can host yourself.
+It's designed be to be minimal, fast, and easy to set up using Docker.
 
 The name comes from:
 - *link* which is often used as a synonym for URLs and bookmarks in common language
-- *Ding* which is german for *thing*
+- *Ding* which is German for thing
-- ...so basically some thing for managing your links
+- ...so basically something for managing your links
 
 **Feature Overview:**
 - Tags for organizing bookmarks
@@ -15,7 +35,7 @@ The name comes from:
 - Bookmark archive
 - Dark mode
 - Automatically creates snapshots of bookmarked websites on [the Internet Archive Wayback Machine](https://archive.org/web/)
-- Automatically provides titles and descriptions of bookmarked websites 
+- Automatically provides titles and descriptions of bookmarked websites
 - Import and export bookmarks in Netscape HTML format
 - Extensions for [Firefox](https://addons.mozilla.org/de/firefox/addon/linkding-extension/) and [Chrome](https://chrome.google.com/webstore/detail/linkding-extension/beakmhbijpdhipnjhnclmhgjlddhidpe), and a bookmarklet that should work in most browsers
 - REST API for developing 3rd party apps
@@ -31,95 +51,84 @@ The name comes from:
 
 ## Installation
 
-The easiest way to run linkding is to use [Docker](https://docs.docker.com/get-started/).  The Docker image is compatible with ARM platforms, so it can be run on a Raspberry Pi.
+linkding is designed to be run with container solutions like [Docker](https://docs.docker.com/get-started/).  The Docker image is compatible with ARM platforms, so it can be run on a Raspberry Pi.
 
-There is also the option to set up the installation manually which I do not support, but can give some pointers on below.
+###  Using Docker
 
-###  Docker setup
+To install linkding using Docker you can just run the [latest image](https://hub.docker.com/repository/docker/sissbruecker/linkding) from Docker Hub:
-
+```shell
-To install linkding using Docker you can just run the image from the Docker registry:
-```
 docker run --name linkding -p 9090:9090 -d sissbruecker/linkding:latest
 ```
-By default the application runs on port `9090`, but you can map it to a different host port by modifying the command above.
+By default, the application runs on port `9090`, you can map it to a different host port by modifying the port mapping in the command above. If everything completed successfully, the application should now be running and can be accessed at http://localhost:9090, provided you did not change the port mapping.
 
-However for **production use** you also want to mount a data folder on your system, so that the applications database is not stored in the container, but on your hosts file system. This is safer in case something happens to the container and makes it easier to update the container later on, or to run backups. To do so you can use the following extended command, where you replace `{host-data-folder}` with the absolute path to a folder on your system where you want to store the data:
+Note that the command above will store the linkding SQLite database in the container, which means that deleting the container, for example when upgrading the installation, will also remove the database. For hosting an actual installation you usually want to store the database on the host system, rather than in the container. To do so, run the following command, and replace the `{host-data-folder}` placeholder with an absolute path to a folder on your host system where you want to store the linkding database:
 ```shell
 docker run --name linkding -p 9090:9090 -v {host-data-folder}:/etc/linkding/data -d sissbruecker/linkding:latest
 ```
 
-If everything completed successfully the application should now be running and can be accessed at http://localhost:9090, provided you did not change the port mapping. 
+To upgrade the installation to a new version, remove the existing container, pull the latest version of the linkding Docker image, and then start a new container using the same command that you used above. There is a [shell script](https://github.com/sissbruecker/linkding/blob/master/install-linkding.sh) available to automate these steps. The script can be configured using environment variables, or you can just modify it.
 
-### Automated Docker setup
+To complete the setup, you still have to [create an initial user](#user-setup), so that you can access your installation.
 
-If you are using a Linux system you can use the following [shell script](https://github.com/sissbruecker/linkding/blob/master/install-linkding.sh) for an automated setup. The script does basically everything described above, but also handles updating an existing container to a new application version (technically replaces the existing container with a new container built from a newer image, while mounting the same data folder).
+###  Using Docker Compose
 
-The script can be configured using shell variables - for more details have a look at the script itself.
+To install linkding using [Docker Compose](https://docs.docker.com/compose/), you can use the [`docker-compose.yml`](https://github.com/sissbruecker/linkding/blob/master/docker-compose.yml) file. Copy the [`.env.sample`](https://github.com/sissbruecker/linkding/blob/master/.env.sample) file to `.env`, configure the parameters, and then run:
-
-###  Docker-compose setup
-
-To install linkding using docker-compose you can use the `docker-compose.yml` file. Copy the `.env.sample` file to `.env` and set your parameters, then run:
 ```shell
 docker-compose up -d
 ```
 
+To complete the setup, you still have to [create an initial user](#user-setup), so that you can access your installation.
+
 ### User setup
 
-Finally you need to create a user so that you can access the application. Replace the credentials in the following command and run it:
+For security reasons, the linkding Docker image does not provide an initial user, so you have to create one after setting up an installation. To do so, replace the credentials in the following command and run it:
 
 **Docker**
 ```shell
 docker exec -it linkding python manage.py createsuperuser --username=joe --email=joe@example.com
 ```
 
-**Docker-compose**
+**Docker Compose**
 ```shell
 docker-compose exec linkding python manage.py createsuperuser --username=joe --email=joe@example.com
 ```
 
 The command will prompt you for a secure password. After the command has completed you can start using the application by logging into the UI with your credentials.
 
-### Manual setup
+### Managed Hosting Options
 
-If you can not or don't want to use Docker you can install the application manually on your server. To do so you can basically follow the steps from the *Development* section below while cross-referencing the `Dockerfile` and `bootstrap.sh` on how to make the application production-ready.
+Self-hosting web applications on your own hardware (unfortunately) still requires a lot of technical know-how, and commitment to maintenance, with regard to keeping everything up-to-date and secure. This can be a huge entry barrier for people who are interested in self-hosting linkding, but lack the technical knowledge to do so. This section is intended to provide alternatives in form of managed hosting solutions. Note that these options are usually commercial offerings, that require paying a (usually monthly) fee for the convenience of being managed by another party. The technical knowledge required to make use of individual options is going to vary, and no guarantees can be made that every option is accessible for everyone. That being said, I hope this section helps in making the application accessible to a wider audience.
 
-### Hosting
+- [linkding on fly.io](https://github.com/fspoettel/linkding-on-fly) - Guide for hosting a linkding installation on [fly.io](https://fly.io). By [fspoettel](https://github.com/fspoettel)
+- [PikaPods.com](https://www.pikapods.com/) - Managed hosting for linkding, EU and US regions available. [1-click setup link](https://www.pikapods.com/pods?run=linkding)
 
-The application runs in a web-server called [uWSGI](https://uwsgi-docs.readthedocs.io/en/latest/) that is production-ready and that you can expose to the web. If you don't know how to configure your server to expose the application to the web there are several more steps involved. I can not support the process here, but I can give some pointers on what to search for:
+## Documentation
-- first get the app running (described in this document)
-- open the port that the application is running on in your servers firewall
-- depending on your network configuration, forward the opened port in your network router, so that the application can be addressed from the internet using your public IP address and the opened port
 
-## Options
+| Document                                                                                        | Description                                              |
+|-------------------------------------------------------------------------------------------------|----------------------------------------------------------|
+| [Options](https://github.com/sissbruecker/linkding/blob/master/docs/Options.md)                 | Lists available options, and describes how to apply them |
+| [Backups](https://github.com/sissbruecker/linkding/blob/master/docs/backup.md)                  | How to backup the linkding database                      |
+| [Troubleshooting](https://github.com/sissbruecker/linkding/blob/master/docs/troubleshooting.md) | Advice for troubleshooting common problems               |
+| [How To](https://github.com/sissbruecker/linkding/blob/master/docs/how-to.md)                   | Tips and tricks around using linking                     |
+| [Admin documentation](https://github.com/sissbruecker/linkding/blob/master/docs/Admin.md)       | User documentation for the Admin UI                      |
+| [API documentation](https://github.com/sissbruecker/linkding/blob/master/docs/API.md)           | Documentation for the REST API                           |
 
-Check the [options document](docs/Options.md) on how to configure your linkding installation.
+## Browser Extension
 
-## Administration
+linkding comes with an official browser extension that allows to quickly add bookmarks, and search bookmarks through the browser's address bar. You can get the extension here:
+- [Mozilla Addon Store](https://addons.mozilla.org/de/firefox/addon/linkding-extension/)
+- [Chrome Web Store](https://chrome.google.com/webstore/detail/linkding-extension/beakmhbijpdhipnjhnclmhgjlddhidpe)
 
-Check the [administration document](docs/Admin.md) on how to use the admin app that is bundled with linkding.
+The extension is open-source as well, and can be found [here](https://github.com/sissbruecker/linkding-extension).
 
-## Backups
+## Community
 
-Check the [backups document](docs/backup.md) for options on how to create backups.
+This section lists community projects around using linkding, in alphabetical order. If you have a project that you want to share with the linkding community, feel free to submit a PR to add your project to this section.
 
-## How To
+- [Helm Chart](https://charts.pascaliske.dev/charts/linkding/) Helm Chart for deploying linkding inside a Kubernetes cluster. By [pascaliske](https://github.com/pascaliske)
-
+- [linkding-extension](https://github.com/jeroenpardon/linkding-extension) Chromium compatible extension that wraps the linkding bookmarklet. Tested with Chrome, Edge, Brave. By [jeroenpardon](https://github.com/jeroenpardon)
-Check the [how-to document](docs/how-to.md) for tips and tricks around using linkding.
+- [linkding-injector](https://github.com/Fivefold/linkding-injector) Injects search results from linkding into the sidebar of search pages like google and duckduckgo. Tested with Firefox and Chrome. By [Fivefold](https://github.com/Fivefold)
-
+- [aiolinkding](https://github.com/bachya/aiolinkding) A Python3, async library to interact with the linkding REST API. By [bachya](https://github.com/bachya)
-## API
-
-The application provides a REST API that can be used by 3rd party applications to manage bookmarks. Check the [API docs](docs/API.md) for further information.
-
-## Troubleshooting
-
-**Import fails with `502 Bad Gateway`**
-
-The default timeout for requests is 60 seconds, after which the application server will cancel the request and return the above error.
-Depending on the system that the application runs on, and the number of bookmarks that need to be imported, the import may take longer than the default 60 seconds.
-
-To increase the timeout you can configure the [`LD_REQUEST_TIMEOUT` option](Options.md#LD_REQUEST_TIMEOUT).
-
-Note that any proxy servers that you are running in front of linkding may have their own timeout settings, which are not affected by the variable.
 
 ## Development
 
@@ -165,9 +174,3 @@ Start the Django development server with:
 python3 manage.py runserver
 ```
 The frontend is now available under http://localhost:8000
-
-## Community
-
-- [Helm Chart](https://charts.pascaliske.dev/charts/linkding/) Helm Chart for deploying linkding inside a Kubernetes cluster. By [pascaliske](https://github.com/pascaliske)
-- [linkding-extension](https://github.com/jeroenpardon/linkding-extension) Chromium compatible extension that wraps the linkding bookmarklet. Tested with Chrome, Edge, Brave. By [jeroenpardon](https://github.com/jeroenpardon)
-- [linkding-injector](https://github.com/Fivefold/linkding-injector) Injects search results from linkding into the sidebar of search pages like google and duckduckgo. Tested with Firefox and Chrome. By [Fivefold](https://github.com/Fivefold)

SECURITY.md

@@ -4,7 +4,7 @@
 
 | Version | Supported          |
 | ------- | ------------------ |
-| 1.8.x   | :white_check_mark: |
+| 1.10.x   | :white_check_mark: |
 
 ## Reporting a Vulnerability
 

bookmarks/queries.py

@@ -55,6 +55,12 @@ def _base_bookmarks_query(user: User, query_string: str) -> QuerySet:
             tags__name__iexact=tag_name
         )
 
+    # Untagged bookmarks
+    if query['untagged']:
+        query_set = query_set.filter(
+            tags=None
+        )
+
     # Sort by date added
     query_set = query_set.order_by('-date_added')
 
@@ -90,11 +96,15 @@ def _parse_query_string(query_string):
     keywords = query_string.strip().split(' ')
     keywords = [word for word in keywords if word]
 
-    search_terms = [word for word in keywords if word[0] != '#']
+    search_terms = [word for word in keywords if word[0] != '#' and word[0] != '!']
     tag_names = [word[1:] for word in keywords if word[0] == '#']
     tag_names = unique(tag_names, str.lower)
 
+    # Special search commands
+    untagged = '!untagged' in keywords
+
     return {
         'search_terms': search_terms,
         'tag_names': tag_names,
+        'untagged': untagged,
     }

bookmarks/services/importer.py

@@ -1,13 +1,13 @@
 import logging
 from dataclasses import dataclass
+from typing import List
 
 from django.contrib.auth.models import User
 from django.utils import timezone
 
-from bookmarks.models import Bookmark, parse_tag_string
+from bookmarks.models import Bookmark, Tag, parse_tag_string
 from bookmarks.services import tasks
 from bookmarks.services.parser import parse, NetscapeBookmark
-from bookmarks.services.tags import get_or_create_tags
 from bookmarks.utils import parse_timestamp
 
 logger = logging.getLogger(__name__)
@@ -20,8 +20,39 @@ class ImportResult:
     failed: int = 0
 
 
+class TagCache:
+    def __init__(self, user: User):
+        self.user = user
+        self.cache = dict()
+        # Init cache with all existing tags for that user
+        tags = Tag.objects.filter(owner=user)
+        for tag in tags:
+            self.put(tag)
+
+    def get(self, tag_name: str):
+        tag_name_lowercase = tag_name.lower()
+        if tag_name_lowercase in self.cache:
+            return self.cache[tag_name_lowercase]
+        else:
+            return None
+
+    def get_all(self, tag_names: List[str]):
+        result = []
+        for tag_name in tag_names:
+            tag = self.get(tag_name)
+            # Prevent returning duplicates
+            if not (tag in result):
+                result.append(tag)
+
+        return result
+
+    def put(self, tag: Tag):
+        self.cache[tag.name.lower()] = tag
+
+
 def import_netscape_html(html: str, user: User):
     result = ImportResult()
+    import_start = timezone.now()
 
     try:
         netscape_bookmarks = parse(html)
@@ -29,26 +60,130 @@ def import_netscape_html(html: str, user: User):
         logging.exception('Could not read bookmarks file.')
         raise
 
+    parse_end = timezone.now()
+    logger.debug(f'Parse duration: {parse_end - import_start}')
+
+    # Create and cache all tags beforehand
+    _create_missing_tags(netscape_bookmarks, user)
+    tag_cache = TagCache(user)
+
+    # Split bookmarks to import into batches, to keep memory usage for bulk operations manageable
+    batches = _get_batches(netscape_bookmarks, 200)
+    for batch in batches:
+        _import_batch(batch, user, tag_cache, result)
+
+    # Create snapshots for newly imported bookmarks
+    tasks.schedule_bookmarks_without_snapshots(user)
+
+    end = timezone.now()
+    logger.debug(f'Import duration: {end - import_start}')
+
+    return result
+
+
+def _create_missing_tags(netscape_bookmarks: List[NetscapeBookmark], user: User):
+    tag_cache = TagCache(user)
+    tags_to_create = []
+
+    for netscape_bookmark in netscape_bookmarks:
+        tag_names = parse_tag_string(netscape_bookmark.tag_string)
+        for tag_name in tag_names:
+            tag = tag_cache.get(tag_name)
+            if not tag:
+                tag = Tag(name=tag_name, owner=user)
+                tag.date_added = timezone.now()
+                tags_to_create.append(tag)
+
+    Tag.objects.bulk_create(tags_to_create)
+
+
+def _get_batches(items: List, batch_size: int):
+    batches = []
+    offset = 0
+    num_items = len(items)
+
+    while offset < num_items:
+        batch = items[offset:min(offset + batch_size, num_items)]
+        if len(batch) > 0:
+            batches.append(batch)
+        offset = offset + batch_size
+
+    return batches
+
+
+def _import_batch(netscape_bookmarks: List[NetscapeBookmark], user: User, tag_cache: TagCache, result: ImportResult):
+    # Query existing bookmarks
+    batch_urls = [bookmark.href for bookmark in netscape_bookmarks]
+    existing_bookmarks = Bookmark.objects.filter(owner=user, url__in=batch_urls)
+
+    # Create or update bookmarks from parsed Netscape bookmarks
+    bookmarks_to_create = []
+    bookmarks_to_update = []
+
     for netscape_bookmark in netscape_bookmarks:
         result.total = result.total + 1
         try:
-            _import_bookmark_tag(netscape_bookmark, user)
+            # Lookup existing bookmark by URL, or create new bookmark if there is no bookmark for that URL yet
+            bookmark = next(
+                (bookmark for bookmark in existing_bookmarks if bookmark.url == netscape_bookmark.href), None)
+            if not bookmark:
+                bookmark = Bookmark(owner=user)
+                is_update = False
+            else:
+                is_update = True
+            # Copy data from parsed bookmark
+            _copy_bookmark_data(netscape_bookmark, bookmark)
+            # Validate bookmark fields, exclude owner to prevent n+1 database query,
+            # also there is no specific validation on owner
+            bookmark.clean_fields(exclude=['owner'])
+            # Schedule for update or insert
+            if is_update:
+                bookmarks_to_update.append(bookmark)
+            else:
+                bookmarks_to_create.append(bookmark)
+
             result.success = result.success + 1
         except:
             shortened_bookmark_tag_str = str(netscape_bookmark)[:100] + '...'
             logging.exception('Error importing bookmark: ' + shortened_bookmark_tag_str)
             result.failed = result.failed + 1
 
-    # Create snapshots for newly imported bookmarks
+    # Bulk update bookmarks in DB
-    tasks.schedule_bookmarks_without_snapshots(user)
+    Bookmark.objects.bulk_update(bookmarks_to_update,
+                                 ['url', 'date_added', 'date_modified', 'unread', 'title', 'description', 'owner'])
+    # Bulk insert new bookmarks into DB
+    Bookmark.objects.bulk_create(bookmarks_to_create)
 
-    return result
+    # Bulk assign tags
+    # In Django 3, bulk_create does not return the auto-generated IDs when bulk inserting,
+    # so we have to reload the inserted bookmarks, and match them to the parsed bookmarks by URL
+    existing_bookmarks = Bookmark.objects.filter(owner=user, url__in=batch_urls)
+
+    BookmarkToTagRelationShip = Bookmark.tags.through
+    relationships = []
+
+    for netscape_bookmark in netscape_bookmarks:
+        # Lookup bookmark by URL again
+        bookmark = next(
+            (bookmark for bookmark in existing_bookmarks if bookmark.url == netscape_bookmark.href), None)
+
+        if not bookmark:
+            # Something is wrong, we should have just created this bookmark
+            shortened_bookmark_tag_str = str(netscape_bookmark)[:100] + '...'
+            logging.warning(
+                f'Failed to assign tags to the bookmark: {shortened_bookmark_tag_str}. Could not find bookmark by URL.')
+
+        # Get tag models by string, schedule inserts for bookmark -> tag associations
+        tag_names = parse_tag_string(netscape_bookmark.tag_string)
+        tags = tag_cache.get_all(tag_names)
+        for tag in tags:
+            relationships.append(BookmarkToTagRelationShip(bookmark=bookmark, tag=tag))
+
+    # Insert all bookmark -> tag associations at once, should ignore errors if association already exists
+    BookmarkToTagRelationShip.objects.bulk_create(relationships, ignore_conflicts=True)
 
 
-def _import_bookmark_tag(netscape_bookmark: NetscapeBookmark, user: User):
+def _copy_bookmark_data(netscape_bookmark: NetscapeBookmark, bookmark: Bookmark):
-    # Either modify existing bookmark for the URL or create new one
-    bookmark = _get_or_create_bookmark(netscape_bookmark.href, user)
-
     bookmark.url = netscape_bookmark.href
     if netscape_bookmark.date_added:
         bookmark.date_added = parse_timestamp(netscape_bookmark.date_added)
@@ -56,24 +191,7 @@ def _import_bookmark_tag(netscape_bookmark: NetscapeBookmark, user: User):
         bookmark.date_added = timezone.now()
     bookmark.date_modified = bookmark.date_added
     bookmark.unread = False
-    bookmark.title = netscape_bookmark.title
+    if netscape_bookmark.title:
+        bookmark.title = netscape_bookmark.title
     if netscape_bookmark.description:
         bookmark.description = netscape_bookmark.description
-    bookmark.owner = user
-
-    bookmark.full_clean()
-    bookmark.save()
-
-    # Set tags
-    tag_names = parse_tag_string(netscape_bookmark.tag_string)
-    tags = get_or_create_tags(tag_names, user)
-
-    bookmark.tags.set(tags)
-    bookmark.save()
-
-
-def _get_or_create_bookmark(url: str, user: User):
-    try:
-        return Bookmark.objects.get(url=url, owner=user)
-    except Bookmark.DoesNotExist:
-        return Bookmark()

bookmarks/services/parser.py

@@ -1,6 +1,6 @@
 from dataclasses import dataclass
-
+from html.parser import HTMLParser
-import pyparsing as pp
+from typing import Dict, List
 
 
 @dataclass
@@ -12,60 +12,72 @@ class NetscapeBookmark:
     tag_string: str
 
 
-def extract_bookmark_link(tag):
+class BookmarkParser(HTMLParser):
-    href = tag[0].href
+    def __init__(self):
-    title = tag[0].text
+        super().__init__()
-    tag_string = tag[0].tags
+        self.bookmarks = []
-    date_added = tag[0].add_date
 
-    return {
+        self.current_tag = None
-        'href': href,
+        self.bookmark = None
-        'title': title,
+        self.href = ''
-        'tag_string': tag_string,
+        self.add_date = ''
-        'date_added': date_added
+        self.tags = ''
-    }
+        self.title = ''
+        self.description = ''
 
+    def handle_starttag(self, tag: str, attrs: list):
+        name = 'handle_start_' + tag.lower()
+        if name in dir(self):
+            getattr(self, name)({k.lower(): v for k, v in attrs})
+        self.current_tag = tag
 
-def extract_bookmark(tag):
+    def handle_endtag(self, tag: str):
-    link = tag[0].link
+        name = 'handle_end_' + tag.lower()
-    description = tag[0].description
+        if name in dir(self):
-    description = description[0] if description else ''
+            getattr(self, name)()
+        self.current_tag = None
 
-    return {
+    def handle_data(self, data):
-        'link': link,
+        name = f'handle_{self.current_tag}_data'
-        'description': description,
+        if name in dir(self):
-    }
+            getattr(self, name)(data)
 
+    def handle_end_dl(self):
+        self.add_bookmark()
 
-def extract_description(tag):
+    def handle_start_dt(self, attrs: Dict[str, str]):
-    return tag[0].strip()
+        self.add_bookmark()
 
-
+    def handle_start_a(self, attrs: Dict[str, str]):
-# define grammar
+        vars(self).update(attrs)
-dt_start, _ = pp.makeHTMLTags("DT")
+        self.bookmark = NetscapeBookmark(
-dd_start, _ = pp.makeHTMLTags("DD")
+            href=self.href,
-a_start, a_end = pp.makeHTMLTags("A")
+            title='',
-bookmark_link_tag = pp.Group(a_start + a_start.tag_body("text") + a_end.suppress())
+            description='',
-bookmark_link_tag.addParseAction(extract_bookmark_link)
+            date_added=self.add_date,
-bookmark_description_tag = dd_start.suppress() + pp.SkipTo(pp.anyOpenTag | pp.anyCloseTag)("description")
+            tag_string=self.tags,
-bookmark_description_tag.addParseAction(extract_description)
-bookmark_tag = pp.Group(dt_start + bookmark_link_tag("link") + pp.ZeroOrMore(bookmark_description_tag)("description"))
-bookmark_tag.addParseAction(extract_bookmark)
-
-
-def parse(html: str) -> [NetscapeBookmark]:
-    matches = bookmark_tag.searchString(html)
-    bookmarks = []
-
-    for match in matches:
-        bookmark_match = match[0]
-        bookmark = NetscapeBookmark(
-            href=bookmark_match['link']['href'],
-            title=bookmark_match['link']['title'],
-            description=bookmark_match['description'],
-            tag_string=bookmark_match['link']['tag_string'],
-            date_added=bookmark_match['link']['date_added'],
         )
-        bookmarks.append(bookmark)
 
-    return bookmarks
+    def handle_a_data(self, data):
+        self.title = data.strip()
+
+    def handle_dd_data(self, data):
+        self.description = data.strip()
+
+    def add_bookmark(self):
+        if self.bookmark:
+            self.bookmark.title = self.title
+            self.bookmark.description = self.description
+            self.bookmarks.append(self.bookmark)
+        self.bookmark = None
+        self.href = ''
+        self.add_date = ''
+        self.tags = ''
+        self.title = ''
+        self.description = ''
+
+
+def parse(html: str) -> List[NetscapeBookmark]:
+    parser = BookmarkParser()
+    parser.feed(html)
+    return parser.bookmarks

bookmarks/services/website_loader.py

@@ -34,7 +34,8 @@ def load_website_metadata(url: str):
 
 
 def load_page(url: str):
-    r = requests.get(url, timeout=10)
+    headers = fake_request_headers()
+    r = requests.get(url, timeout=10, headers=headers)
 
     # Use charset_normalizer to determine encoding that best matches the response content
     # Several sites seem to specify the response encoding incorrectly, so we ignore it and use custom logic instead
@@ -42,3 +43,13 @@ def load_page(url: str):
     # before trying to determine one
     results = from_bytes(r.content)
     return str(results.best())
+
+
+def fake_request_headers():
+    return {
+        "Accept": "text/html,application/xhtml+xml,application/xml",
+        "Accept-Encoding": "gzip, deflate",
+        "Dnt": "1",
+        "Upgrade-Insecure-Requests": "1",
+        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.0.0 Safari/537.36",
+    }

bookmarks/styles/util.scss

@@ -15,3 +15,7 @@
 .text-gray-dark {
   color: $gray-color-dark;
 }
+
+.align-baseline {
+  align-items: baseline;
+}

bookmarks/templates/bookmarks/archive.html

@@ -33,8 +33,10 @@
 
         {# Tag list #}
         <section class="content-area column col-4 hide-md">
-            <div class="content-area-header">
+            <div class="content-area-header align-baseline">
                 <h2>Tags</h2>
+                <div class="spacer"></div>
+                <a href="?q=!untagged" class="text-gray text-sm">Show Untagged</a>
             </div>
             {% tag_cloud tags %}
         </section>

bookmarks/templates/bookmarks/index.html

@@ -33,8 +33,10 @@
 
         {# Tag list #}
         <section class="content-area column col-4 hide-md">
-            <div class="content-area-header">
+            <div class="content-area-header align-baseline">
                 <h2>Tags</h2>
+                <div class="spacer"></div>
+                <a href="?q=!untagged" class="text-gray text-sm">Show Untagged</a>
             </div>
             {% tag_cloud tags %}
         </section>

bookmarks/templates/bookmarks/layout.html

@@ -30,12 +30,15 @@
 <header>
     {% if has_toasts %}
     <div class="toasts container grid-lg">
+      <form action="{% url 'bookmarks:toasts.acknowledge' %}?return_url={{ request.path | urlencode }}" method="post">
+        {% csrf_token %}
         {% for toast in toast_messages %}
             <div class="toast">
                 {{ toast.message }}
-                <a href="{% url 'bookmarks:toasts.acknowledge' toast.id %}?return_url={{ request.path | urlencode }}" class="btn btn-clear float-right"></a>
+                <button type="submit" name="toast" value="{{ toast.id }}" class="btn btn-clear float-right"></button>
             </div>
         {% endfor %}
+      </form>
     </div>
     {% endif %}
     <div class="navbar container grid-lg">

bookmarks/tests/helpers.py

@@ -1,5 +1,7 @@
 import random
 import logging
+from dataclasses import dataclass
+from typing import Optional, List
 
 from django.contrib.auth.models import User
 from django.utils import timezone
@@ -87,6 +89,42 @@ class LinkdingApiTestCase(APITestCase):
         return response
 
 
+class BookmarkHtmlTag:
+    def __init__(self, href: str = '', title: str = '', description: str = '', add_date: str = '', tags: str = ''):
+        self.href = href
+        self.title = title
+        self.description = description
+        self.add_date = add_date
+        self.tags = tags
+
+
+class ImportTestMixin:
+    def render_tag(self, tag: BookmarkHtmlTag):
+        return f'''
+        <DT>
+        <A {f'HREF="{tag.href}"' if tag.href else ''}
+           {f'ADD_DATE="{tag.add_date}"' if tag.add_date else ''}
+           {f'TAGS="{tag.tags}"' if tag.tags else ''}>
+           {tag.title if tag.title else ''}
+        </A>
+        {f'<DD>{tag.description}' if tag.description else ''}
+        '''
+
+    def render_html(self, tags: List[BookmarkHtmlTag] = None, tags_html: str = ''):
+        if tags:
+            rendered_tags = [self.render_tag(tag) for tag in tags]
+            tags_html = '\n'.join(rendered_tags)
+        return f'''
+        <!DOCTYPE NETSCAPE-Bookmark-file-1>
+        <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">
+        <TITLE>Bookmarks</TITLE>
+        <H1>Bookmarks</H1>
+        <DL><p>
+        {tags_html}
+        </DL><p>
+        '''
+
+
 _words = [
     'quasi',
     'consequatur',

bookmarks/tests/test_bookmark_index_view.py

@@ -156,3 +156,8 @@ class BookmarkIndexViewTestCase(TestCase, BookmarkFactoryMixin):
         response = self.client.get(reverse('bookmarks:index'))
 
         self.assertVisibleBookmarks(response, visible_bookmarks, '_self')
+
+    def test_should_show_link_for_untagged_bookmarks(self):
+        response = self.client.get(reverse('bookmarks:index'))
+
+        self.assertContains(response, '<a href="?q=!untagged" class="text-gray text-sm">Show Untagged</a>')

bookmarks/tests/test_importer.py

@@ -1,29 +1,204 @@
+from typing import List
 from unittest.mock import patch
 
-from django.test import TestCase
+from django.test import TestCase, override_settings
+from django.utils import timezone
 
-from bookmarks.models import Tag
+from bookmarks.models import Bookmark, Tag, parse_tag_string
 from bookmarks.services import tasks
 from bookmarks.services.importer import import_netscape_html
-from bookmarks.tests.helpers import BookmarkFactoryMixin, disable_logging
+from bookmarks.tests.helpers import BookmarkFactoryMixin, ImportTestMixin, BookmarkHtmlTag, disable_logging
+from bookmarks.utils import parse_timestamp
 
 
-class ImporterTestCase(TestCase, BookmarkFactoryMixin):
+class ImporterTestCase(TestCase, BookmarkFactoryMixin, ImportTestMixin):
 
-    def create_import_html(self, bookmark_tags_string: str):
+    def assertBookmarksImported(self, html_tags: List[BookmarkHtmlTag]):
-        return f'''
+        for html_tag in html_tags:
-<!DOCTYPE NETSCAPE-Bookmark-file-1>
+            bookmark = Bookmark.objects.get(url=html_tag.href)
-<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">
+            self.assertIsNotNone(bookmark)
-<TITLE>Bookmarks</TITLE>
+
-<H1>Bookmarks</H1>
+            self.assertEqual(bookmark.title, html_tag.title)
-<DL><p>
+            self.assertEqual(bookmark.description, html_tag.description)
-{bookmark_tags_string}
+            self.assertEqual(bookmark.date_added, parse_timestamp(html_tag.add_date))
-</DL><p>
+
-        '''
+            tag_names = parse_tag_string(html_tag.tags)
+
+            # Check assigned tags
+            for tag_name in tag_names:
+                tag = next(
+                    (tag for tag in bookmark.tags.all() if tag.name == tag_name), None)
+                self.assertIsNotNone(tag)
+
+    def test_import(self):
+        html_tags = [
+            BookmarkHtmlTag(href='https://example.com', title='Example title', description='Example description',
+                            add_date='1', tags='example-tag'),
+            BookmarkHtmlTag(href='https://foo.com', title='Foo title', description='',
+                            add_date='2', tags=''),
+            BookmarkHtmlTag(href='https://bar.com', title='Bar title', description='Bar description',
+                            add_date='3', tags='bar-tag, other-tag'),
+        ]
+        import_html = self.render_html(tags=html_tags)
+        result = import_netscape_html(import_html, self.get_or_create_test_user())
+
+        # Check result
+        self.assertEqual(result.total, 3)
+        self.assertEqual(result.success, 3)
+        self.assertEqual(result.failed, 0)
+
+        # Check bookmarks
+        bookmarks = Bookmark.objects.all()
+        self.assertEqual(len(bookmarks), 3)
+        self.assertBookmarksImported(html_tags)
+
+    def test_synchronize(self):
+        # Initial import
+        html_tags = [
+            BookmarkHtmlTag(href='https://example.com', title='Example title', description='Example description',
+                            add_date='1', tags='example-tag'),
+            BookmarkHtmlTag(href='https://foo.com', title='Foo title', description='',
+                            add_date='2', tags=''),
+            BookmarkHtmlTag(href='https://bar.com', title='Bar title', description='Bar description',
+                            add_date='3', tags='bar-tag, other-tag'),
+        ]
+        import_html = self.render_html(tags=html_tags)
+        import_netscape_html(import_html, self.get_or_create_test_user())
+
+        # Change data, add some new data
+        html_tags = [
+            BookmarkHtmlTag(href='https://example.com', title='Updated Example title',
+                            description='Updated Example description', add_date='111', tags='updated-example-tag'),
+            BookmarkHtmlTag(href='https://foo.com', title='Updated Foo title', description='Updated Foo description',
+                            add_date='222', tags='new-tag'),
+            BookmarkHtmlTag(href='https://bar.com', title='Updated Bar title', description='Updated Bar description',
+                            add_date='333', tags='updated-bar-tag, updated-other-tag'),
+            BookmarkHtmlTag(href='https://baz.com', add_date='444', tags='baz-tag')
+        ]
+
+        # Import updated data
+        import_html = self.render_html(tags=html_tags)
+        result = import_netscape_html(import_html, self.get_or_create_test_user())
+
+        # Check result
+        self.assertEqual(result.total, 4)
+        self.assertEqual(result.success, 4)
+        self.assertEqual(result.failed, 0)
+
+        # Check bookmarks
+        bookmarks = Bookmark.objects.all()
+        self.assertEqual(len(bookmarks), 4)
+        self.assertBookmarksImported(html_tags)
+
+    def test_import_with_some_invalid_bookmarks(self):
+        html_tags = [
+            BookmarkHtmlTag(href='https://example.com'),
+            # Invalid URL
+            BookmarkHtmlTag(href='foo.com'),
+            # No URL
+            BookmarkHtmlTag(),
+        ]
+        import_html = self.render_html(tags=html_tags)
+        result = import_netscape_html(import_html, self.get_or_create_test_user())
+
+        # Check result
+        self.assertEqual(result.total, 3)
+        self.assertEqual(result.success, 1)
+        self.assertEqual(result.failed, 2)
+
+        # Check bookmarks
+        bookmarks = Bookmark.objects.all()
+        self.assertEqual(len(bookmarks), 1)
+        self.assertBookmarksImported(html_tags[1:1])
+
+    def test_import_tags(self):
+        html_tags = [
+            BookmarkHtmlTag(href='https://example.com', tags='tag1'),
+            BookmarkHtmlTag(href='https://foo.com', tags='tag2'),
+            BookmarkHtmlTag(href='https://bar.com', tags='tag3'),
+        ]
+        import_html = self.render_html(tags=html_tags)
+        import_netscape_html(import_html, self.get_or_create_test_user())
+
+        self.assertEqual(Tag.objects.count(), 3)
+
+    def test_create_missing_tags(self):
+        html_tags = [
+            BookmarkHtmlTag(href='https://example.com', tags='tag1'),
+            BookmarkHtmlTag(href='https://foo.com', tags='tag2'),
+            BookmarkHtmlTag(href='https://bar.com', tags='tag3'),
+        ]
+        import_html = self.render_html(tags=html_tags)
+        import_netscape_html(import_html, self.get_or_create_test_user())
+
+        html_tags.append(
+            BookmarkHtmlTag(href='https://baz.com', tags='tag4')
+        )
+        import_html = self.render_html(tags=html_tags)
+        import_netscape_html(import_html, self.get_or_create_test_user())
+
+        self.assertEqual(Tag.objects.count(), 4)
+
+    def test_should_append_tags_to_bookmark_when_reimporting_with_different_tags(self):
+        html_tags = [
+            BookmarkHtmlTag(href='https://example.com', tags='tag1'),
+        ]
+        import_html = self.render_html(tags=html_tags)
+        import_netscape_html(import_html, self.get_or_create_test_user())
+
+        html_tags.append(
+            BookmarkHtmlTag(href='https://example.com', tags='tag2, tag3')
+        )
+        import_html = self.render_html(tags=html_tags)
+        import_netscape_html(import_html, self.get_or_create_test_user())
+
+        self.assertEqual(Bookmark.objects.count(), 1)
+        self.assertEqual(Bookmark.objects.all()[0].tags.all().count(), 3)
+
+    @override_settings(USE_TZ=False)
+    def test_use_current_date_when_no_add_date(self):
+        test_html = self.render_html(tags_html=f'''
+            <DT><A HREF="https://example.com">Example.com</A>
+            <DD>Example.com
+        ''')
+
+        with patch.object(timezone, 'now', return_value=timezone.datetime(2021, 1, 1)):
+            import_netscape_html(test_html, self.get_or_create_test_user())
+
+            self.assertEqual(Bookmark.objects.count(), 1)
+            self.assertEqual(Bookmark.objects.all()[0].date_added, timezone.datetime(2021, 1, 1))
+
+    def test_keep_title_if_imported_bookmark_has_empty_title(self):
+        test_html = self.render_html(tags=[
+            BookmarkHtmlTag(href='https://example.com', title='Example.com')
+        ])
+        import_netscape_html(test_html, self.get_or_create_test_user())
+
+        test_html = self.render_html(tags=[
+            BookmarkHtmlTag(href='https://example.com')
+        ])
+        import_netscape_html(test_html, self.get_or_create_test_user())
+
+        self.assertEqual(Bookmark.objects.count(), 1)
+        self.assertEqual(Bookmark.objects.all()[0].title, 'Example.com')
+
+    def test_keep_description_if_imported_bookmark_has_empty_description(self):
+        test_html = self.render_html(tags=[
+            BookmarkHtmlTag(href='https://example.com', description='Example.com')
+        ])
+        import_netscape_html(test_html, self.get_or_create_test_user())
+
+        test_html = self.render_html(tags=[
+            BookmarkHtmlTag(href='https://example.com')
+        ])
+        import_netscape_html(test_html, self.get_or_create_test_user())
+
+        self.assertEqual(Bookmark.objects.count(), 1)
+        self.assertEqual(Bookmark.objects.all()[0].description, 'Example.com')
 
     def test_replace_whitespace_in_tag_names(self):
-        test_html = self.create_import_html(f'''
+        test_html = self.render_html(tags_html=f'''
-            <DT><A HREF="https://example.com" ADD_DATE="1616337559" PRIVATE="0" TOREAD="0" TAGS="tag 1, tag 2, tag 3">Example.com</A>
+            <DT><A HREF="https://example.com" TAGS="tag 1, tag 2, tag 3">Example.com</A>
             <DD>Example.com
         ''')
         import_netscape_html(test_html, self.get_or_create_test_user())
@@ -35,22 +210,22 @@ class ImporterTestCase(TestCase, BookmarkFactoryMixin):
 
     @disable_logging
     def test_validate_empty_or_missing_bookmark_url(self):
-        test_html = self.create_import_html(f'''
+        test_html = self.render_html(tags_html=f'''
-            <!-- Empty URL -->
+            <DT><A HREF="">Empty URL</A>
-            <DT><A HREF="" ADD_DATE="1616337559" PRIVATE="0" TOREAD="0" TAGS="tag3">Empty URL</A>
             <DD>Empty URL
-            <!-- Missing URL -->
+            <DT><A>Missing URL</A>
-            <DT><A ADD_DATE="1616337559" PRIVATE="0" TOREAD="0" TAGS="tag3">Missing URL</A>
             <DD>Missing URL
         ''')
 
         import_result = import_netscape_html(test_html, self.get_or_create_test_user())
 
+        self.assertEqual(Bookmark.objects.count(), 0)
         self.assertEqual(import_result.success, 0)
+        self.assertEqual(import_result.failed, 2)
 
     def test_schedule_snapshot_creation(self):
         user = self.get_or_create_test_user()
-        test_html = self.create_import_html('')
+        test_html = self.render_html(tags_html='')
 
         with patch.object(tasks, 'schedule_bookmarks_without_snapshots') as mock_schedule_bookmarks_without_snapshots:
             import_netscape_html(test_html, user)

bookmarks/tests/test_parser.py

@@ -0,0 +1,122 @@
+from typing import List
+
+from django.test import TestCase
+
+from bookmarks.services.parser import NetscapeBookmark
+from bookmarks.services.parser import parse
+from bookmarks.tests.helpers import ImportTestMixin, BookmarkHtmlTag
+
+
+class ParserTestCase(TestCase, ImportTestMixin):
+    def assertTagsEqual(self, bookmarks: List[NetscapeBookmark], html_tags: List[BookmarkHtmlTag]):
+        self.assertEqual(len(bookmarks), len(html_tags))
+        for bookmark in bookmarks:
+            html_tag = html_tags[bookmarks.index(bookmark)]
+            self.assertEqual(bookmark.href, html_tag.href)
+            self.assertEqual(bookmark.title, html_tag.title)
+            self.assertEqual(bookmark.date_added, html_tag.add_date)
+            self.assertEqual(bookmark.description, html_tag.description)
+            self.assertEqual(bookmark.tag_string, html_tag.tags)
+
+    def test_parse_bookmarks(self):
+        html_tags = [
+            BookmarkHtmlTag(href='https://example.com', title='Example title', description='Example description',
+                            add_date='1', tags='example-tag'),
+            BookmarkHtmlTag(href='https://foo.com', title='Foo title', description='',
+                            add_date='2', tags=''),
+            BookmarkHtmlTag(href='https://bar.com', title='Bar title', description='Bar description',
+                            add_date='3', tags='bar-tag, other-tag'),
+        ]
+        html = self.render_html(html_tags)
+        bookmarks = parse(html)
+
+        self.assertTagsEqual(bookmarks, html_tags)
+
+    def test_no_bookmarks(self):
+        html = self.render_html()
+        bookmarks = parse(html)
+
+        self.assertEqual(bookmarks, [])
+
+    def test_reset_properties_after_adding_bookmark(self):
+        html_tags = [
+            BookmarkHtmlTag(href='https://example.com', title='Example title', description='Example description',
+                            add_date='1', tags='example-tag'),
+            BookmarkHtmlTag(href='', title='', description='',
+                            add_date='', tags='')
+        ]
+        html = self.render_html(html_tags)
+        bookmarks = parse(html)
+
+        self.assertTagsEqual(bookmarks, html_tags)
+
+    def test_empty_title(self):
+        html_tags = [
+            BookmarkHtmlTag(href='https://example.com', title='', description='Example description',
+                            add_date='1', tags='example-tag'),
+        ]
+        html = self.render_html(tags_html='''
+        <DT><A HREF="https://example.com" ADD_DATE="1" TAGS="example-tag"></A>
+        <DD>Example description
+        ''')
+        bookmarks = parse(html)
+
+        self.assertTagsEqual(bookmarks, html_tags)
+
+    def test_with_closing_description_tag(self):
+        html_tags = [
+            BookmarkHtmlTag(href='https://example.com', title='Example title', description='Example description',
+                            add_date='1', tags='example-tag'),
+            BookmarkHtmlTag(href='https://foo.com', title='Foo title', description='',
+                            add_date='2', tags=''),
+        ]
+        html = self.render_html(tags_html='''
+        <DT><A HREF="https://example.com" ADD_DATE="1" TAGS="example-tag">Example title</A>
+        <DD>Example description</DD>
+        <DT><A HREF="https://foo.com" ADD_DATE="2">Foo title</A>
+        <DD></DD>
+        ''')
+        bookmarks = parse(html)
+
+        self.assertTagsEqual(bookmarks, html_tags)
+
+    def test_description_tag_before_anchor_tag(self):
+        html_tags = [
+            BookmarkHtmlTag(href='https://example.com', title='Example title', description='Example description',
+                            add_date='1', tags='example-tag'),
+            BookmarkHtmlTag(href='https://foo.com', title='Foo title', description='',
+                            add_date='2', tags=''),
+        ]
+        html = self.render_html(tags_html='''
+        <DT><DD>Example description</DD>
+        <A HREF="https://example.com" ADD_DATE="1" TAGS="example-tag">Example title</A>
+        <DT><DD></DD>
+        <A HREF="https://foo.com" ADD_DATE="2">Foo title</A>
+        ''')
+        bookmarks = parse(html)
+
+        self.assertTagsEqual(bookmarks, html_tags)
+
+    def test_with_folders(self):
+        html_tags = [
+            BookmarkHtmlTag(href='https://example.com', title='Example title', description='Example description',
+                            add_date='1', tags='example-tag'),
+            BookmarkHtmlTag(href='https://foo.com', title='Foo title', description='',
+                            add_date='2', tags=''),
+        ]
+        html = self.render_html(tags_html='''
+        <DL><p>
+            <DT><H3>Folder 1</H3>
+            <DL><p>
+                <DT><A HREF="https://example.com" ADD_DATE="1" TAGS="example-tag">Example title</A>
+                <DD>Example description
+            </DL><p>
+            <DT><H3>Folder 2</H3>
+            <DL><p>
+                <DT><A HREF="https://foo.com" ADD_DATE="2">Foo title</A>
+            </DL><p>
+        </DL><p>
+        ''')
+        bookmarks = parse(html)
+
+        self.assertTagsEqual(bookmarks, html_tags)

bookmarks/tests/test_queries.py

@@ -289,6 +289,60 @@ class QueriesTestCase(TestCase, BookmarkFactoryMixin):
             self.assertEqual(bookmark.tag_string, None)
             self.assertTrue(bookmark.tag_projection)
 
+    def test_query_bookmarks_untagged_should_return_untagged_bookmarks_only(self):
+        tag = self.setup_tag()
+        untagged_bookmark = self.setup_bookmark()
+        self.setup_bookmark(tags=[tag])
+        self.setup_bookmark(tags=[tag])
+
+        query = queries.query_bookmarks(self.user, '!untagged')
+        self.assertCountEqual(list(query), [untagged_bookmark])
+
+    def test_query_bookmarks_untagged_should_be_combinable_with_search_terms(self):
+        tag = self.setup_tag()
+        untagged_bookmark = self.setup_bookmark(title='term1')
+        self.setup_bookmark(title='term2')
+        self.setup_bookmark(tags=[tag])
+
+        query = queries.query_bookmarks(self.user, '!untagged term1')
+        self.assertCountEqual(list(query), [untagged_bookmark])
+
+    def test_query_bookmarks_untagged_should_not_be_combinable_with_tags(self):
+        tag = self.setup_tag()
+        self.setup_bookmark()
+        self.setup_bookmark(tags=[tag])
+        self.setup_bookmark(tags=[tag])
+
+        query = queries.query_bookmarks(self.user, f'!untagged #{tag.name}')
+        self.assertCountEqual(list(query), [])
+
+    def test_query_archived_bookmarks_untagged_should_return_untagged_bookmarks_only(self):
+        tag = self.setup_tag()
+        untagged_bookmark = self.setup_bookmark(is_archived=True)
+        self.setup_bookmark(is_archived=True, tags=[tag])
+        self.setup_bookmark(is_archived=True, tags=[tag])
+
+        query = queries.query_archived_bookmarks(self.user, '!untagged')
+        self.assertCountEqual(list(query), [untagged_bookmark])
+
+    def test_query_archived_bookmarks_untagged_should_be_combinable_with_search_terms(self):
+        tag = self.setup_tag()
+        untagged_bookmark = self.setup_bookmark(is_archived=True, title='term1')
+        self.setup_bookmark(is_archived=True, title='term2')
+        self.setup_bookmark(is_archived=True, tags=[tag])
+
+        query = queries.query_archived_bookmarks(self.user, '!untagged term1')
+        self.assertCountEqual(list(query), [untagged_bookmark])
+
+    def test_query_archived_bookmarks_untagged_should_not_be_combinable_with_tags(self):
+        tag = self.setup_tag()
+        self.setup_bookmark(is_archived=True)
+        self.setup_bookmark(is_archived=True, tags=[tag])
+        self.setup_bookmark(is_archived=True, tags=[tag])
+
+        query = queries.query_archived_bookmarks(self.user, f'!untagged #{tag.name}')
+        self.assertCountEqual(list(query), [])
+
     def test_query_bookmark_tags_should_return_all_tags_for_empty_query(self):
         self.setup_tag_search_data()
 
@@ -460,3 +514,35 @@ class QueriesTestCase(TestCase, BookmarkFactoryMixin):
         query = queries.query_archived_bookmark_tags(self.user, '')
 
         self.assertQueryResult(query, [self.get_tags_from_bookmarks(owned_bookmarks)])
+
+    def test_query_bookmark_tags_untagged_should_never_return_any_tags(self):
+        tag = self.setup_tag()
+        self.setup_bookmark()
+        self.setup_bookmark(title='term1')
+        self.setup_bookmark(title='term1', tags=[tag])
+        self.setup_bookmark(tags=[tag])
+
+        query = queries.query_bookmark_tags(self.user, '!untagged')
+        self.assertCountEqual(list(query), [])
+
+        query = queries.query_bookmark_tags(self.user, '!untagged term1')
+        self.assertCountEqual(list(query), [])
+
+        query = queries.query_bookmark_tags(self.user, f'!untagged #{tag.name}')
+        self.assertCountEqual(list(query), [])
+
+    def test_query_archived_bookmark_tags_untagged_should_never_return_any_tags(self):
+        tag = self.setup_tag()
+        self.setup_bookmark(is_archived=True)
+        self.setup_bookmark(is_archived=True, title='term1')
+        self.setup_bookmark(is_archived=True, title='term1', tags=[tag])
+        self.setup_bookmark(is_archived=True, tags=[tag])
+
+        query = queries.query_archived_bookmark_tags(self.user, '!untagged')
+        self.assertCountEqual(list(query), [])
+
+        query = queries.query_archived_bookmark_tags(self.user, '!untagged term1')
+        self.assertCountEqual(list(query), [])
+
+        query = queries.query_archived_bookmark_tags(self.user, f'!untagged #{tag.name}')
+        self.assertCountEqual(list(query), [])

bookmarks/tests/test_toasts_view.py

@@ -60,12 +60,20 @@ class ToastsViewTestCase(TestCase, BookmarkFactoryMixin):
         # Should not render toasts
         self.assertContains(response, '<div class="toast">', count=0)
 
+    def test_form_tag(self):
+        self.create_toast()
+        expected_form_tag = f'<form action="{reverse("bookmarks:toasts.acknowledge")}?return_url={reverse("bookmarks:index")}" method="post">'
+
+        response = self.client.get(reverse('bookmarks:index'))
+
+        self.assertContains(response, expected_form_tag)
+
     def test_toast_content(self):
         toast = self.create_toast()
         expected_toast = f'''
             <div class="toast">
                 {toast.message}
-                <a href="{reverse('bookmarks:toasts.acknowledge', args=[toast.id])}?return_url={reverse('bookmarks:index')}" class="btn btn-clear float-right"></a>
+                <button type="submit" name="toast" value="{toast.id}" class="btn btn-clear float-right"></button>
             </div>        
         '''
 
@@ -77,7 +85,9 @@ class ToastsViewTestCase(TestCase, BookmarkFactoryMixin):
     def test_acknowledge_toast(self):
         toast = self.create_toast()
 
-        self.client.get(reverse('bookmarks:toasts.acknowledge', args=[toast.id]))
+        self.client.post(reverse('bookmarks:toasts.acknowledge'), {
+            'toast': [toast.id],
+        })
 
         toast.refresh_from_db()
         self.assertTrue(toast.acknowledged)
@@ -85,17 +95,21 @@ class ToastsViewTestCase(TestCase, BookmarkFactoryMixin):
     def test_acknowledge_toast_should_redirect_to_return_url(self):
         toast = self.create_toast()
         return_url = reverse('bookmarks:settings.general')
-        acknowledge_url = reverse('bookmarks:toasts.acknowledge', args=[toast.id])
+        acknowledge_url = reverse('bookmarks:toasts.acknowledge')
         acknowledge_url = acknowledge_url + '?return_url=' + return_url
 
-        response = self.client.get(acknowledge_url)
+        response = self.client.post(acknowledge_url, {
+            'toast': [toast.id],
+        })
 
         self.assertRedirects(response, return_url)
 
     def test_acknowledge_toast_should_redirect_to_index_by_default(self):
         toast = self.create_toast()
 
-        response = self.client.get(reverse('bookmarks:toasts.acknowledge', args=[toast.id]))
+        response = self.client.post(reverse('bookmarks:toasts.acknowledge'), {
+            'toast': [toast.id],
+        })
 
         self.assertRedirects(response, reverse('bookmarks:index'))
 
@@ -104,5 +118,7 @@ class ToastsViewTestCase(TestCase, BookmarkFactoryMixin):
         other_user = User.objects.create_user('otheruser', 'otheruser@example.com', 'password123')
         toast = self.create_toast(user=other_user)
 
-        response = self.client.get(reverse('bookmarks:toasts.acknowledge', args=[toast.id]))
+        response = self.client.post(reverse('bookmarks:toasts.acknowledge'), {
+            'toast': [toast.id],
+        })
         self.assertEqual(response.status_code, 404)

bookmarks/urls.py

@@ -23,7 +23,7 @@ urlpatterns = [
     path('settings/import', views.settings.bookmark_import, name='settings.import'),
     path('settings/export', views.settings.bookmark_export, name='settings.export'),
     # Toasts
-    path('toasts/<int:toast_id>/acknowledge', views.toasts.acknowledge, name='toasts.acknowledge'),
+    path('toasts/acknowledge', views.toasts.acknowledge, name='toasts.acknowledge'),
     # API
     path('api/', include(router.urls), name='api')
 ]

bookmarks/views/toasts.py

@@ -7,7 +7,8 @@ from bookmarks.utils import get_safe_return_url
 
 
 @login_required
-def acknowledge(request, toast_id: int):
+def acknowledge(request):
+    toast_id = request.POST['toast']
     try:
         toast = Toast.objects.get(pk=toast_id, owner=request.user)
     except Toast.DoesNotExist:

docs/header.svg

@@ -0,0 +1,41 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg width="100%" height="100%" viewBox="0 0 2599 591" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xml:space="preserve" xmlns:serif="http://www.serif.com/" style="fill-rule:evenodd;clip-rule:evenodd;stroke-linejoin:round;stroke-miterlimit:1.5;">
+    <g transform="matrix(1.18075,0,0,1.18075,-1265.31,-1395.82)">
+        <circle cx="1314.98" cy="1424.52" r="190.496" style="fill:rgb(88,86,224);"/>
+    </g>
+    <g transform="matrix(1,0,0,1,-1017.49,-1140.55)">
+        <g transform="matrix(0.707351,0.706862,-0.706862,0.707351,1331.93,-512.804)">
+            <path d="M1244.39,1293.95L1244.39,1493.59C1244.39,1493.59 1243.58,1561.48 1319.29,1562.47C1395.27,1563.46 1394.17,1493.59 1394.17,1493.59L1394.17,1293.95" style="fill:none;stroke:white;stroke-width:31.25px;"/>
+        </g>
+        <g transform="matrix(-0.710067,-0.704134,0.704134,-0.710067,1284.12,3366.41)">
+            <path d="M1244.39,1293.95L1244.39,1493.59C1244.39,1493.59 1243.58,1561.48 1319.29,1562.47C1395.27,1563.46 1394.17,1493.59 1394.17,1493.59L1394.17,1293.95" style="fill:none;stroke:white;stroke-width:31.25px;"/>
+        </g>
+    </g>
+    <g transform="matrix(8.26174,0,0,8.26174,-5762.21,-2037.46)">
+        <g transform="matrix(50,0,0,50,770.835,299.13)">
+            <path d="M0.078,-0.714L0.078,-0L0.551,-0L0.551,-0.08L0.173,-0.08L0.173,-0.714L0.078,-0.714Z" style="fill:rgb(132,132,228);fill-rule:nonzero;"/>
+        </g>
+        <g transform="matrix(50,0,0,50,798.635,299.13)">
+            <rect x="0.082" y="-0.714" width="0.095" height="0.714" style="fill:rgb(132,132,228);fill-rule:nonzero;"/>
+        </g>
+        <g transform="matrix(50,0,0,50,811.585,299.13)">
+            <path d="M0.077,-0.714L0.077,-0L0.167,-0L0.167,-0.573L0.169,-0.573L0.542,-0L0.646,-0L0.646,-0.714L0.556,-0.714L0.556,-0.135L0.554,-0.135L0.178,-0.714L0.077,-0.714Z" style="fill:rgb(132,132,228);fill-rule:nonzero;"/>
+        </g>
+        <g transform="matrix(50,0,0,50,847.685,299.13)">
+            <path d="M0.078,-0.714L0.078,-0L0.173,-0L0.173,-0.25L0.292,-0.361L0.55,-0L0.67,-0L0.357,-0.426L0.658,-0.714L0.535,-0.714L0.173,-0.358L0.173,-0.714L0.078,-0.714Z" style="fill:rgb(132,132,228);fill-rule:nonzero;"/>
+        </g>
+        <g transform="matrix(50,0,0,50,881.035,299.13)">
+            <path d="M0.173,-0.08L0.173,-0.634L0.333,-0.634C0.377,-0.634 0.414,-0.628 0.444,-0.616C0.474,-0.603 0.498,-0.585 0.518,-0.562C0.537,-0.538 0.55,-0.509 0.559,-0.476C0.567,-0.442 0.571,-0.404 0.571,-0.361C0.571,-0.317 0.567,-0.28 0.558,-0.249C0.549,-0.218 0.537,-0.192 0.523,-0.171C0.509,-0.15 0.493,-0.134 0.476,-0.122C0.458,-0.11 0.44,-0.101 0.422,-0.095C0.404,-0.088 0.387,-0.084 0.371,-0.083C0.355,-0.081 0.342,-0.08 0.331,-0.08L0.173,-0.08ZM0.078,-0.714L0.078,-0L0.323,-0C0.382,-0 0.434,-0.008 0.477,-0.025C0.52,-0.042 0.556,-0.066 0.584,-0.098C0.612,-0.129 0.633,-0.168 0.646,-0.215C0.659,-0.261 0.666,-0.314 0.666,-0.374C0.666,-0.489 0.636,-0.574 0.577,-0.63C0.518,-0.686 0.433,-0.714 0.323,-0.714L0.078,-0.714Z" style="fill:rgb(132,132,228);fill-rule:nonzero;"/>
+        </g>
+        <g transform="matrix(50,0,0,50,916.235,299.13)">
+            <rect x="0.082" y="-0.714" width="0.095" height="0.714" style="fill:rgb(132,132,228);fill-rule:nonzero;"/>
+        </g>
+        <g transform="matrix(50,0,0,50,929.185,299.13)">
+            <path d="M0.077,-0.714L0.077,-0L0.167,-0L0.167,-0.573L0.169,-0.573L0.542,-0L0.646,-0L0.646,-0.714L0.556,-0.714L0.556,-0.135L0.554,-0.135L0.178,-0.714L0.077,-0.714Z" style="fill:rgb(132,132,228);fill-rule:nonzero;"/>
+        </g>
+        <g transform="matrix(50,0,0,50,965.285,299.13)">
+            <path d="M0.612,-0.089L0.637,-0L0.697,-0L0.697,-0.376L0.384,-0.376L0.384,-0.296L0.612,-0.296C0.613,-0.263 0.609,-0.233 0.599,-0.205C0.589,-0.176 0.574,-0.152 0.555,-0.131C0.535,-0.11 0.511,-0.093 0.482,-0.081C0.453,-0.069 0.42,-0.063 0.383,-0.063C0.343,-0.063 0.308,-0.071 0.278,-0.087C0.247,-0.102 0.222,-0.123 0.201,-0.15C0.18,-0.176 0.165,-0.206 0.154,-0.241C0.143,-0.275 0.138,-0.311 0.138,-0.348C0.138,-0.386 0.143,-0.423 0.152,-0.46C0.161,-0.496 0.176,-0.528 0.196,-0.557C0.215,-0.585 0.241,-0.608 0.272,-0.625C0.303,-0.642 0.34,-0.651 0.383,-0.651C0.41,-0.651 0.435,-0.648 0.459,-0.642C0.482,-0.635 0.503,-0.626 0.522,-0.613C0.541,-0.6 0.556,-0.584 0.569,-0.565C0.582,-0.545 0.59,-0.521 0.595,-0.494L0.69,-0.494C0.683,-0.536 0.671,-0.572 0.653,-0.602C0.634,-0.631 0.612,-0.656 0.585,-0.675C0.558,-0.694 0.527,-0.708 0.493,-0.718C0.458,-0.727 0.422,-0.731 0.383,-0.731C0.326,-0.731 0.277,-0.721 0.235,-0.7C0.192,-0.679 0.157,-0.65 0.129,-0.615C0.1,-0.58 0.079,-0.539 0.065,-0.492C0.05,-0.444 0.043,-0.395 0.043,-0.343C0.043,-0.296 0.051,-0.251 0.066,-0.208C0.081,-0.165 0.104,-0.126 0.133,-0.093C0.162,-0.06 0.198,-0.033 0.24,-0.014C0.282,0.006 0.33,0.016 0.383,0.016C0.425,0.016 0.467,0.008 0.508,-0.009C0.549,-0.025 0.584,-0.052 0.612,-0.089Z" style="fill:rgb(132,132,228);fill-rule:nonzero;"/>
+        </g>
+    </g>
+</svg>

docs/how-to.md

@@ -44,7 +44,7 @@ This how-to explains how to make use of the app shortcuts iOS app to create a sh
 - create new shortcut
 - go to shortcut details, enable to option to show the shortcut in share menu
 - from the available share input types only select "URL"
-- add Safari action "Display website in Safari" (paraphrasing, not sure how it's called in english)
+- add Safari action "Show Web Page At"
 - for URL enter your linkding instance URL and specifically point to the new bookmark form, then add the shortcut input variable from the list of suggested variables after the URL parameter. Visually it should look something like this: `https://linkding.mydomain.com/bookmarks/new?url=[Shortcut input]`, where `[Shortcut input]` is a visual block that was inserted after selecting the shortcut input variable suggestion. This is basically a placeholder that will get replaced with the actual URL that you want to bookmark. See screenshot at the end for an example on how this looks.
 - save, give the shortcut a nice name + glyph
 

docs/troubleshooting.md

@@ -0,0 +1,10 @@
+# Troubleshooting
+
+## Import fails with `502 Bad Gateway`
+
+The default timeout for requests is 60 seconds, after which the application server will cancel the request and return the above error.
+Depending on the system that the application runs on, and the number of bookmarks that need to be imported, the import may take longer than the default 60 seconds.
+
+To increase the timeout you can configure the [`LD_REQUEST_TIMEOUT` option](Options.md#LD_REQUEST_TIMEOUT).
+
+Note that any proxy servers that you are running in front of linkding may have their own timeout settings, which are not affected by the variable.

package.json

@@ -1,6 +1,6 @@
 {
   "name": "linkding",
-  "version": "1.9.0",
+  "version": "1.10.1",
   "description": "",
   "main": "index.js",
   "scripts": {

requirements.prod.txt

@@ -13,7 +13,6 @@ django-sass-processor==1.0.1
 django-widget-tweaks==1.4.8
 djangorestframework==3.12.4
 idna==2.8
-pyparsing==2.4.7
 python-dateutil==2.8.1
 pytz==2021.1
 requests==2.26.0

requirements.txt

@@ -18,7 +18,6 @@ django-widget-tweaks==1.4.8
 djangorestframework==3.12.4
 idna==2.8
 libsass==0.21.0
-pyparsing==2.4.7
 python-dateutil==2.8.1
 pytz==2021.1
 rcssmin==1.0.6

siteroot/settings/dev.py

@@ -48,6 +48,11 @@ LOGGING = {
             'level': 'DEBUG',
             'handlers': ['console'],
             'propagate': False,
+        },
+        'bookmarks.services.importer': {  # Log importer debug output
+            'level': 'DEBUG',
+            'handlers': ['console'],
+            'propagate': False,
         }
     }
 }

uwsgi.ini

@@ -3,7 +3,7 @@ chdir = /etc/linkding
 module = siteroot.wsgi:application
 env = DJANGO_SETTINGS_MODULE=siteroot.settings.prod
 static-map = /static=static
-processes = 4
+processes = 2
 threads = 2
 pidfile = /tmp/linkding.pid
 vacuum=True

version.txt

@@ -1 +1 @@
-1.9.0
+1.10.1